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The tasks described in this report were performed 
by Educational Testing Service (ETS) pursuant to Contract 
Number OEC-72-4450 with the United States Department of 
Health, Education, and Welfare, Office of Education (OE) , 
dated May 26, 1972. The activities of the initiating 
agency in OE, the National Center for Educational Commu- 
nication, became part of the National Institute of Edu- 
cation after its inception in the summer of 1972. 



Abstract 



The 1972-1973 Panel Review of Products (PROP) was conducted for the 
National Institute of Education (NIE) to gather descriptive and evalua- 
tive information needed to help in developing dissemination plans for 
selected educational products. Tasks included organization of a 
nationally representative review Panel, composition of an initial pool 
of developer-nominated educational products on which verification and 
cost dataware available, completion of product descriptions highlight- 
ing information of significance for comparative evaluation purposes, 
development of a criterion base for product review and analysis, and 
selection by the Panel of a limited pool of products to be subjected to 
further study in the context of the dissemination strategies of NIE. 
Specifically, the PROP mission was to select from an initial pool of 90 
educational products a smaller pool of 15 to 35 products on which 
dissemination recommendaticus could be made and from which NIE might 
choose products for its dissemination programs as they evolve. 

prop's pool at one point contained 190 products from some 50 insti- 
tutions engaged in product development. Preliminary screening decisions 
by NIE narrowed the field so that 90 products from 33 institutions were 
designated as the pool to be exposed to PROF evaluation procedures. Of 
that pool, 23 products representing four research and development centers, 
four regional laboratories, and six other institutions received favorable 
PROP Panel votes. These products became the subjects of detailed 
dissemination recommendations made by the Panel to NIE. An additional 25 
products from four centers, five laboratories, and six other institutions 
were voted by the Panel to have sufficient promise to be categorized as 
"hold for subsequent review." On the 42 remaining products, the Panel 
took no positive action for a variety of reasons. 

This year's review Panel of ten members, of whom half had served with 
the PEP Panel (ETS , 19 71) and half were newly appointed, met twice within 
a 5-week span. All members were present at all sessions. Staff support 
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was provided by Educational Testing Service (ETS). The advance preparations 
and follow-up activities required an ETS project team of five — who share 
authorship of this report — as well as 33 other members of the professional 
staff, drawn from three divisions, to communicate with principal investi- 
gators and to carry the burden of individual product review and analysis. 

Although the criterion base for Panel reviei^ of products in PROP was 
quite similar to the one used in PEP 1971 — goals, effectiveness, adopt- 
ability, and costs — the decision rules for product selection were dramati- 
cally different. In thumbnail sketch, the procedures were as follows: 
after criteria were applied in the study of a given product and resulting 
overall individual judgments were made on all of the products in the 
initial pool, the Panel then moved to plenary consideration of products 
that appeared likely to receive favorable ratings. In the case of each 
such product, several panelists were in agreement that it was at least in 
the "3 - Will accept" category on a five-level rating scale used to sum- 
marize overall individual ratings. Thorough discussions were held on each 
product, followed by a Panel vote. A product at this point could be voted 
"recommended" or "not recommended." The procedural difference this year 
was in the multiple basis upon which a product could receive a "recom- 
mended" vote, essentially in any of four variations: 

o Def ini tely recommended for dissemination 

o Recommended for dissemination if an identified 
condition is satisfied 

o Recommended for dissemination action concurrent 
with other activities indicated 

o Recommended for dissemination action following 
other activities indicated. 

The latitude thus afforded in the decision-making process allowed 
Panel dissemination recommendations to be made with respect to a number 
of promising educational products where some lack in development or eval- 
uation identified by the Panel would otherwise have prohibited definitive 
actions from having been taken. As a consequence, the PROP Panel had the 
opportunity concurrently to select exemplary products and to advise on 
further development and evaluation steps judged to be needed in the course 
of moving products toward widespread use. This appears to be a marked 
advancement in the state of the art of dissemination-oriented evaluation. 

-ii- 



Foreword 



This is the second comparative review of educational products that 
has been conducted for the purpose of advancing the dissemination- 
diffusion programs of the National Center for Educational Communication 
(NCEC) and represents one of the first operational activities of the 
National Institute of Education (NIE) , The activity, designated the 
Panel Review of Products (PROP) , moved into its active stage during the 
period when the enabling legislation for NIE was before the Congress. 
Monitoring of the project was nx)ved from The United States Office of 
Education (OE) to NIE during the first month of NIE operation. ' As a 
consequence, the policies that served to guide the ^asks pursued in the 
accomplishment of the mission, especially in the early months, were 
evolved on an ad hoc basis as circumstances required, sometimes on a 
day-to-day basis. During the period of NIE's gestation, the intensity 
and effectiveness of cooperation by OE and NIE staff and by product 
developers and their institutions in the field were exemplary, as were 
the flexibility, energy levels, and quality of work of PROP panelists 
and ETS staff members. 

As with the Project to Evaluate Products (PEP) a year earlier, PROP 
was devoted to the comparative evaluation of educational products well 
on the way in their respective development cycles and to preparation of 
substantive recommendations related to the dissemination of the more 
exemplary products among them. In this frame of reference, educational 
products are things OE funds have 'iDOUght" — the outcomes of educational 
res exarch and development accomplished in large measure as OE-funded 
activities. In addition to the programs monitored in pre-NIE days by 
the National Center for Educational Research and Development (NCERD) — 
the sole source of products for the 19 70-19 71 PEP pool, PROP also drew 
products for the 19 72-19 73 pool from programs supported by OE's Bureau 
for the Education of the Handicapped (BEH) . It is to be hoped that the 
next comparative evaluation done to assist in disseminating improved 
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educational, practices will cast an even larger net to capture In the pool 
for pane] review another year developmental products coming out of programs 
sp'jnsored by t:he National Science Foundation and otlier foundations^ as well 
as by other government and private agencies serving education. 

IL is LTiportant to note that the subjects of PROF evaluatit)n are 
products . not the prog rams or pro j ects out of which they were born. llie 
information base, the criteria applied, and the judgments made have all 
been product-oriented. T!ie rules of the game for PROP would apply, in a 
sense, reasonably well in an educational consumers' movement, looking hard- 
est as the rules do at the question, '^vUiat products are most worthy of 
widest use in our school^;?" The caveat here is that although a criterion 
base served to anchor the PROP evaluation process, anC thus tempered the 
selections, the criteria used were, after all, compatible with the 
diss emination- di f f us ion interests of ME — and thus were quite explicit* 
To use this evaluation paradigm for other purposes would require subscrib- 
ing to a siniilar criterion formulat i on--goal-bound , l:ieavily dependent on 
ef f ectivenes., measures, and centrally concerned with both dollar and non- 
dollar cos ts . 

It seems reasonable to expect tliat the 19 72-1973 PROP activity has 
moved the state of the art of product evaluation to a level at which 
developers will be able to do more self-selection of products appropriate 
for entry into subsequent PROP-type pools. Certainly, many of the prod- 
ucts entered this time should not be entered again. One outcome that, it 
is to be hoped, will be ^^timulated by tills report is a needed set of guide- 
lines that will assist developers in differentiating products that would 
be likely to benefit from ceniralized dissemination strategies from products 
that are better served through dissemination carried out on a decentralized 
basis, as 3 for example, an integrated researcVi-development-evaluat ion- 
diffusion model suggested by a systems approach to educational research and 
development. The suggested guidelines, to be effective, would invite entry 
into a PROP pool of only those products for which an NIE dissemination 
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strategy had been formulated. In short, products for nomination to tlu^ next 
PROP pool should be seen by their developers as being best served bv NIK 
dissemination and as being of sufficient current strength to suggest such 
action. 

Wesley W. Walton 

Educational Testing Service 
Princeton, New J e rs ey 

April 1973 
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Introduction 



The Panel Review of Products (PROP) for 1972-1973 was conducted on a 
pool of 90 educational products. For 23, the ten-member Panel submitted 
dissemination- related recommendations; an additional 25 products were 
nominated for review in another year. For the 42 remaining, there were 
neither dissemination recommendations nor suggestions concerning 
subsequent review. The present volume describes in detail the procedures 
followed in product review by the Panel, while a companion volume gives 
a summary of Panel recommendations on and a nontechnical description of 
each of the 23 exemplary products which received dissemination-related 
recommendations . 
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Perspectives or^ Panel Review of 
Educational troducts 

The 19 72-19 73 Panel Review of Products (PROP) was a natural outgrowth 
of an earlier effort, the Product Evaluation Project (PEP) carried out by 
Educational Testing Service (ETS) in 1971 f or \he National Center for 
Educational Communication (NCEC) of the United States Office of Education 
(USOE) , The 19 71 project represented a new type of endeavor, directed at 
identifying outstanding educational products and focusing their dissemi- 
nation in such a way that they could be quickly and effectively installed 
in the nation^s schools. The discussion of PEP that follows emphasizes 
the similarities between PEP and PROP, while calling specific attention to 
those facets in which the two projects differed. 

The field of products from which the PEP panel nominated outstanding 
examples came from the regional educational laboratories, research and 
development centers, and a number of universities and was limited to 
entries which had been developed under the auspices of the National Center 
for Educational Research and Development (NCERD) in the USOE. After various 
criteria, principally the availability of adequate validation data, had 
been applied to an initial pool of 70 products, some 51 products became the 
subjects of panel selection deliberations. PROP, by contrast, worked fT;om 
a larger field of product nominators, including, in addition to the sources 
utilized for PEP, such sources as the Bureau for the Education of the 
Handicapped and the list of recommended products which had been generated 
in the so-called Impact Study conducted by the American Institutes for 
Research in 19 71. The PROP pool in the first months of the National 
Institute of Education (NIE) gradually bL'ilt up to a much larger number of 
products — almost 200 — but Wcis finally reduced by various administrative 
decisions to a field of 90, from which the Panel was asked to select some 
15 to 35 to be recommended for dissemination. 

For PEP a panel of eight appraisers wprked to select exemplary educa- 
tional products. Th^y included curriculum specialists, evaluators, school 
administrators, teachers of teachers, measurement specialists, and critics 
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of current educational research and development efforts. For PROP the 
number of panelists was increased to ten, in part to accommodate the much 
larger number of products to be reviewed, but mainly in order to include 
two additional viewpoints, that of a classroom teacher and that of a 
rejearch person in a large- city school system. 

The .fEP panel met three times, whereas the PROP panelists completed 
their proUucc review in two, albeit harried, two-day meetings. This 
difference can be accounted for in large part by the fact that PEP was a 
pioneering effort and a large proportion of the panel's meeting time wa^ 
necessarily devoted to the arduous task of working out in great detail 
the criteria and mechanisms for product review and- s election . Because of 
the firm base established by the PEP panel in this aspect of the project 
and because of the helpful suggestions provided to future panels for 
improvements, the PROP Panel, relying in substantial measure on ETS staff 
support, was able to proceed most expeditiously through this part of its 
work and to move almost immediately (during the first morning of its first 
meeting) to the actual task of product appraisal. 

Both the PEP and the PROP panels found it expedient to split into two 
subpanels in order to accomplish a substantial portion of their work. In 
both cases, however, concerted efforts were made to ensure that the two 
subpanels were operating by the same groundrules and in each instance all 
final decisions were made in plenary session, so that no panelist was ever 
deprived of the opportunity to plead either for or against any given product. 

The ultimate mission of the two panels did differ somewhat, in that 
pep's task was to actually nominate a small group of recommended "finalists" 
from among its total product pool. The^PROP goal, on the other hand, was 
to nominate a much larger group of exemplary products , from among which the 
NIE would be free to draw in implementing its larger policies for educational 
prodTict dissemination. In actual fact, the PEP panel recommended nine 
products for dissemination focus. Of these, five were designated as being 
first priority selections and four were marked as second priority. PROP, 
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by contrast, nominated a much larger number, 15, for actual dissemination 
funding i with an additional eight products conditionally recommended. That 
is to say, the Panel urged that funds be expended for such purposes as 
additional field trials, independent evaluations, or even further develop- 
ment in the belief that these eight products were of sufficient promise as 
to be likely to merit dissemination recommendations after certain 
deficiences had been remedied. The PEP panel, in addition, selected from 
among its final pool of 51 products nine which they recommended for deferral 
to 1972, largely because adequate evaluation and validation data were not 
yet fully available. A second group of nine was also suggested for deferral, 
on a somewhat less optimistic and lower-priority basis. The PROP Panel 
selected, from its pool of 90 products, 25 which it labeled "Hold" in the 
belief that they were of sufficient promise to merit another look by some 
future panel after such things as additional information, validity data, 
and final reports from developers became available. 

Both PEP and PROP functioned in much the same way in that they were 
able to develop and x/crk with selection criteria that could be applied 
across an eclectic assemblage of products applicable to a variety of 
educational levels — from preschool to adult, in':end;id for diverse socio- 
cultural populations, covering a broad range of academic fields and cross- 
disciplines, intended to serve a broad array of educational purposes, and 
utilizing the gamut of hardware and softSA/are techniques. 

Both the PEP and the PROP panelists saw it as a vital part of their 
mission to make recommendations for future product review panels which 
would enhance their operations and make them more efficient. The PEP 
panel did its work so effectively that the PROP group vere enabled to, as 
it were, **stand on the shoulders of giants." Their entire operation was 
made smoother and swifter in large part as a result of the work laid out 
by PEP and its suggestions for future procedures. The PROP Panel, in turn, 
felt an obligation to analyze its experience in such a way that similar 
activities in the future would be more effective. They enlarged the scope 
of their recommendations, however, to include fundamental policy consider- 
ations. These, if implemented, should enable the government to expend its 
available dissemination funds wisely and well. 



Appointment of the 
1972-1973 Panel for Review of Products 

Introduction . Acceptance of responsibility for key decisions by a top- 
flight group of devoted although disinterested individuals representative 
of cha.ige agents, curriculiim planners, and materials users is critical 
to the success of a project such as the Panel Review of Products. The 
predecessor PEP panel ±n 1971 had recommended an increase in panel size 
from eight to ten, with the two added members being a research worker 
from a public school system and a practicing elementary or secondary 
school classroom teacher. A first order of business for PROP thus became 
nomination and, after NIE approval, appointment of a review Panel, taking 
into account the 19 71 Panel^s recommendations. 

As in 1971, the PROP Panel was to include curriculum specialists, 
evaluators, teacher educators, and measurement specialists. Some of 
those appointed were to be members of special target populations for 
which new products had been developed. A special effort was made to 
include some of the sternest critics of educational research and develop- 
ment in the United States and some who had participated in carrying out 
the programs of educational research sponsored by the United States Office 
cf Education. By including those known to be critical, assurance of 
credibility for the evaluations made by the Panel would be achieved. Those 
who had been involved during the formative period of government-sponsored 
educational research and development would be able to supply judgments as 
to how well the products and other materials now issuing from the 
laboratories, centers, and universities realized the initial expectations 
for them. 

Tae Panel was looked upon as the final decision-making body. Other 
individuals working on the project were seen as sources of staff support 
to make the Panel's tasks in the course of reaching decisions as efficient, 
efi'ective, and pleasant as possible. The Panel's main tasks were to (1) 
establish the criteria for the selection of recommended products and the 
procedures for applying them to the process of product review and to 
dissemination- related decisions, (2) review and analyze all products in 
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the pool and specify additional information needed tL make effective and 
discriminating judgments, (3) select and make appropriaL ^ suggestions 
regarding dissemination- related action on some 15 to 35 pro 'ucts to be 
recommended for focused dissemination by the NIE, and (4) desi^^nate other 
groupings of products and suggest possible courses of action pri(,r to 
their later dissemination review, 

PROP Panel, 19 72-19 73 . It was deemed essential that the ten-member Panel 
appointed should provide direct representation from among the following 
sections: 

o Curriculum and Instruction 

o Teacher Training 

o Evaluation 

o Product Development 

o School Administration 

o Philosophy ar;d History 

o Urban Affairs 

o Lay Criticism of the Public Schools 
o Action-Oriented Educational Research 
o Classroom Teaching 

In attempting to cover so many fields and points of view with so few 
panelists, it was necessary to identify potential panelists whose 
credentials would, in most cases, span several of the categories sought. 
At the same time it was thought desirable to achieve a reasonably good 
geographic distribution. The Panel finally selected did, in fact, fulfill 
both these goals. In effect, each panelist covered several fields. The 
1972-1973 Panel included four women, two members especially sensitive to 
minority-group problems, and one with wide experience in the publishing 
and distribution of educational products. 

The 1971 panel had recommended some holdover from its membership to 
provide continuity; accordingly four of the PROP panelists were appointed 
from among that group, a fifth had been a consultant for the 1971 project, 
and five were newly appointed. 

A list of the 19 72-19 73 PROP Panel members, together with summaries 
of the individual members* credentials, follows. Robert Blanchard served 
id 3 the Panel's informal chairman during its plenary session. 
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Robert Wesley B Ian chard Superintendent of Schools , Portland , Oregon 

1969- 

Formerly Superintendent of Schools, Montclair, New Jersey 

J964-1969 

Assistant Superintendent, South Huntington 
Schools, Long Island, New York 
1962-1964 

Assistant to the Superintendent, Greenwich 
Connecticut 
1960-1962 

High School Principal and Teacher in Maine 

New Jersey Commissioner to the Education 

Commission of the States 
President-Elect, Metropolitan School Study 
Counci 1 

Chairman, Large Cities Superintendents 
Executive Committee, The Council of the Great 
City Schools 

Education A.B. Bowdoin 

Ed.M. University of Maine 
Ed.D. Harvard University 



Other relevan t 
expe rience 



Joan Bollenbacher 



Coordinator , Evaluation and Accountabili ty 
Branch, Cincinnati Public Schools 



Formerly 



With the Cincinnati Public School System si ^ce 
1944: 

Supervisor of the Testing Division 
1949-1960 

Director of the Division of Evaluation Services 
19 60-19 73 



Other relevant 
experience 



National office of the National Council on 
Me as uremer.t in Education and the Association 
for Measurement and Evaluation in Guidance 

Member of Miami University Development Council 



Education 



B. S . Miami Unive rsity (Ohio) 

M.S. University of Wisconsin 

Additional graduate work at Columbia 
U^.-Lversity and the University of 
Cincinnati 
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Edythe J. Gaines 
Fo rmerly 



Other relevant 
experience 



Education 



Coiranuni ty S uperintendent , Communi ty S choo 1 
District 12, Bronx, New York 

Director of the Learning Cooperative for the 
City School District of New York 
19 71-19 72 

Assistant principal and principal, Joan of 
Arc Junior High School, Manhattan 
1960-19 71 

Teacher and curriculum coordinator, New York 
City secondary schools 

Taught courses at Hunter College, Nev^ York 
University > and Teachers College , Columbia 
University 

Consultant to school systems in Maryland and 

Pennsylvania 
Board of Trustees, Montclair State College 
Board of Overseers Visiting Committee, 
Harvard Graduate School of Education 
Board of Directors, Associate Harvard Alumni 
Chairman, Alumni Council, Harvard Graduate 

School of Education 
Board of Advisors, Children's Television Work- 
shop (The Electric Company) 

B.A. Hunter College of the City University 



M.A. 
Ed.D. 



of New York 

New York University 

Harvard Universi ty 
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Kenneth K. Komoski 



President and Director, Educational Products 
Information Exchange Institute 
196 7- 



Fo rmerly 



Senior Program Associate, Institute for 
Educational Development 
1966-1967 

Associate Executive Officer, Institute of 
Educational Technology, Teachers College, 
Columbia University 
1964-1966 

Teacher in independent schools in New Jersey 
and New York 



Other relevant 
experience 



Chai rman , Educa ti onal Technology Fo rum , 19 70 

White House Conference on Children 
Consultant to President's Panel on Education 
Consultant to UNESCO Secretariat for use of 

programmed instruction in teacher education 

in Africa 

1962-1965 

Director, Programmed Instruction Institute, 

University of Ibadan, Nigeria 

Summer 1963 
Director, Programmed Instruction Institute, 

Unive rs i ty of Gh ana , Gh ana 

Summer 1963 
19 71 PEP Subcontractor 



Education 



B.A. Ac^ lia University 

M.A. A. _ aula University 

A.B.D. Columbia University-Union Theological 
Seminary Joint Program in Philosophy 
of Religion 
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David R. Krathwohl 



Dean, School of Education, Syracuse University 
1965- 



Formerly 



Director, Bureau of Educational Research, 
Michigan State University 
1963-1965 

Research Coordinator and Professor, Bur^^^u of 
Educational Research, Michigan State I nlv^' ity 
1955-1963 

Assistant Director, Unit of Evaluation, Bureau 
of Educational Research, University of Illinois 
1949-1955 



Other relevant 
experience 



President, American Educational Research 
Association 
1968 

Chairman, Board of Trustees, Eastern Regional 

las ti tute for Education 
Coauthor of Taxonomy of Educational Objectives , 

The Classification of Educational Goals 
19 71 PEP Panelist 



Education 



B.S . University of Chicago 

M.A. University of Chicago 

Ph.D. University of Chicago 
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Arthur Allen Lumsdaine 



Processor of Psychology and Education and 
Chairman of the Psychology Department^ 
University of Washington 
1965- 



Formerly 



Professor of Education, University of Californin 
at Los Angeles 
1960-1964 

Executive Scientist, American Institutes for 
Research , Pi ttsburgh 
1958-1960 

Research Scientist and Laboratory' Director, 
U.S. Air Force 
1949-195 8 



Other relevant 
experience 



Research adviser, American Institutes for Research 
Editorial consultant for journals in psychology 

and education 
Visiting scientist to UNESCO and nations abroad 
President, APA Division of Educational Psychology 
Author on learning, teaching, and programmed 

ins truction 

Consultant, East-West Center, University of Hawaii 



Education 



B.S. University of Washington 

Ph . D. S tanf ord Univers itv 



Ruth Mancuso 



Member, N.J. State Board of Education - Chairman, 

Committee on Evaluation 
Director, Audiovisual Aids Commission, Camden 

and Gloucester Counties, New Jersey 



Fo rme r ly 



President, National School Boards Association 
President , New Jersey State School Board 
Association 

President, Local School Board, Glassboro, New 
Jersey 

Chairman, N.J. State Committee on School 

Regionalization 
Member, N.J. State Committee on Vocational 

Education 
Public School Teacher 
National Assessment Advisory Panel 
Advisory Commintee for ERIC Clearinghouse on 

Tests and Measurements 
1971 PEP Panelist 



Education 



B.S. Trenton State College 

M.A. Co lumbia University 
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Michael Scriven Professor , Philosophy Department , Univers ity of 

California at Berkeley 
1966- 



Fo rmerly Professor of History and Philosophy of Science, 

Indiana University 
1960-1966 

Fellow, Center for Advanced Study in the 
Behavioral Sciences 
1963 

Taught at Swarthmore College and University of 
Minnesota 
1952-1960 

Whitehead Fellow, Harvard Graduate School of 
Education 
1971 

Other relevant Editorial boards of Encyclopedia of Philosophy , 

experience American Philosophical Quarterly, Metaphilos- 

ophy, and Contemporary Psychology and editorial 

consultant for Science 
Board of Directors, Social Science Education, Inc. 

1963- 

Director, Evaluation of Education Materials 
Project, SSEC 
1963-1967 

Director, SSEC project on role of values in the 
social s tudies 
1963-1966 

Chairman, Advisory Board for Evaluation, Central 

Midwestern Region Educational Laboratory', Inc. 
Advisory Board, Social Studies Program, Minnesota 

Mining and Manufacturing Corporation, Project 

Follow- Through 

1968-1969 

Consultant , U.S. Office of Education 
1968-1969 

Evaluation Consultant, Marin School Board, 

Social Studies Project 

1969- 
19 71 PEP Panelist 



Education A.B. University of Melbourne 

M.A. University of Melbourne 

Ph.D. Oxford University 
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Beverly Trollman 



English and Remedial Reading Teacher, 

Manhattan Junior High School, Manhattan. 
Kans as 



Other relevant 
expe rience 



Contributing editor, "The Hutison News 
Staff member, college newspaper 
Master's thesis, '^icroteaching " 



Education 



B,A. Kansas State University 

M.A. Kansas State University 

Ed.M. Kansas State University 
Doctoral candidate — Kansas State University 
(in curriculum and instruction) 



Bernard Watson Professor and Chairman of Urban Education, 

Temple University 
19 70- 

Formerly Deputy Superintendent for Planning, 

Philadelphia, Pennsylvania School District 
Associate Superintendent for Innovative 

Programs , Philadelphia 
Staff associate for Midwestern administrative 

center at University of Chicago 
Teacher, Counselor, Vice-Principal, and 
Principal in Gary, Indiana 

Local boards for Urban Coalition and Model 

Cities Programs 
Board of trustees for two private schools 
A variety of committees for such things 
as National Teacher Corps and Ford and 
Rockefeller Foundations 
Lecturer at Princeton, Yale, and University of 

Pennsylvania 
1971 PEP Panelist 

Education B.S. Indiana University 

M.A. Unive rsity of Illinois 

Ph.D . University of Chicago 



Other relevant 
experience 
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Recommendations for Future Panels 

Despite the diversity in its membership, the Panel was able to work 
in an exceptionally harmonious fashion and, in its final deliberations, 
achieved a remarkable degree of consensus. Rapport was quickly estab- 
lished at the outset, in part because of the carry-over of five members 
with experience from 1971. Open discussicn and receptivity on the part 
of all panelists to viewpoints which differed from their own made it 
possible for the incisive perceptions of individual panelists who had 
special expertise to be shared fruitfully with the group. 

It would be difficult to imagine a more competent, hard-working, and 
effective panel than the one which served PROP in 1972-1973. '^ince the 
black panelists were able to make a particularly effective and insightful 
contribution in pointing out product strengths and weaknesses for special 
populations, it would seem desirable to attempt to include still other 
minority group members in the future. 

Two recommendations were made by the Panel for similar activities in 
the future. One was that an initial product pool as large as 90 should be 
subjected to an initial screening by at least three or even more subpanels , 
with similar products organized in advance into as many subpanel groupings 
as would seem to be necessary. The other recommendation, somewhat in the 
same vein, was that a plan for splitting the group into subpanels requires 
that more time be made available, or a more effective mechanism arranged, 
for each subpanel to become familiar with the products being reviewed by 
the other subpanels, fso that the ultimate decisions made in plenary sessions 
about specific products and dissemination plans for them can be made at 
the highest possible level of confidence. Both these recommendations are 
central to any future plan to convene a panel for the comparative review 
of educational products. 
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Composition of the Initial Pool 
for Product Review 

Initial steps in the formation of a pool of educational products to be 
reviewed by the 1972-1973 PROP Panel flowed naturally out of the 1971 PEP 
project. Products designated by the PEP panelists to be subjected to future 
review became the nucleus of the PROP pool. To this were added, at the 
request of NIE, the 21 products cited as exemplary in the 1971 study con- 
ducted by the American Institutes for Research (AIR) entitled **The Evaluation 
of the Impact of Educational Research and Development Products." 

In order to secure concurrence from developers on inclusion of 
products from these two sources in the 19 72-19 73 PROP review, and to obtain 
other product nominations for the pool, the letter which begins on the follow- 
ing page was sent to each developer whose product(s) appeared on the initial 
list. The developer was asked whether he or she (a) wished to have the 
product (s) considered by the PROP Panel and (b) had additional products at 
an advanced stage of development to submit. This step produced over 120 
nominations, primarily from regional laboratories and research and de /elop- 
ment centers, but also from colleges and universities and private, 
independent research organizations. To augment the pool generated thus far 
senior staff members working on PROP made several trips to Washington to 
comb the files of NCERD and other agencies within the USOE to locate other 
products which, according to the record, were at an advanced stage of 
development . 

It had been a strong recommendation from the 1971 PEP panel that 
rigorous restrictions be put upon the product nomination process in an 
attempt to prevent the submission of marginal products which had not been 
adequately tested and validated. Thus careful inquiries were made concern- 
ing product verification at every stage of the, search. 

A final source of product nomination was the Bureau for the Education 
of the Handicapped (BEH) which nominated products designed for its special 
audience. Developers of all products identified in the last two steps 
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As you may recall, in 1971 ETS had a contract with the 
National Center for Educational Communication (NCEC) to establish 
a nationally representative Panel and to support the Panel *s 
efforts as it nominated from a pool of verified products a smaller 
pool of products from which NCEC could select those on which to 
concentrate its dissemination efforts. Five products out of the 
nine nominated have since had the benefit of NCEC dissemination 
funding , 

We have a similar contract for jPancl Review £f _Products this 
year and are now assembling the 1972 pool of about 100 products which 
will be subjected to Panel review. The objectives for this yearns 
Panel are to identify from the 1972 pool approxim;itely 25 products 
which the new National Institute of Education will be able to con- 
sider in devising and executing programs related to the dissemination 
and diffusion functions that now fall within its domain, and ETS's 
objectives are to extend our perceptions related to these products 
so that their potentials may be fully understood and described. NIE 
will subsequently select among the products identified those to receive 
the benexit of dissemination funding in the current fiscal year. 

The makeup of the 1972 pool will include (a) products considered 
in 1972 whose developers concur in their inclusion on this round, 

(b) those selected in the American Institutes of Research (AIR) Impact 
Study whose developers concur in their inclusion in our study, and 

(c) other products thought by their originators to have reached that 
stage in their development which would make their inclusion in the 
pool appropriate at this time. 

In general, products to be included should be at an advanced 
stage of development (6 or higher on NCERD^s C-1 Product Development 
Status Report), with field test results available for review by our 
Panels when they meet in November of 1972*. Products already in the 
hands of commercial publishers will not be considered this year. 



The first panel meeting was actually held in January of 1973# 
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Our intention this year during the early stage is to ask 
the developer only for information about his product that is not 
available from other sources, including NIE, PARADE, NCERD, CEDaR, 
TAP, and PEP. Our most critical needs for information on your pro- 
ducts (at least until we see what information is available elsewhere) 
relate to field testing, validation, and verification data that you 
have compiled, analyzed, and interpreted. Such materials of more 
recent date than April 1971 will be welcome additions to our product 
dossiers. We would appreciate your sending now as much documentation 
of this kind as is readily available. Additional materials can be 
added to our product dossiers until November 1972. 

Thank you in advance for filling out the enclosed checklist 
and returning it in the stamped, self-addressed envelope. 

Sincerely, 



Wesley W. Walton 
Program Director 
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mentioned were also sent the letter included earlier to see whether they 
wished a given product to be included and whether they had further entries 
to submit. 

After completion of the step just described, the number of products in 
the working pool had risen to almost 200. Since plans called for the review 
of approximately , 100 products, administrative criteria for cutting the size 
of the pool back to around 100 were worked out jointly by ETS and the NIE • 
One decision made which eliminated a large number of products was to exclude 
all products which were already being distributed commercially, unless these 
products came either from the BEH, one of the regional laboratories or a 
research and development center. 

After application of the NIE-endorsed screening criteria and after 
products had been withdrawn at the request of their developers, the pool of 
products for Panel review stabilized at 90. These are listed, in accession 
number order, in Appendix A. The code number appearing after each product 
name is a classification scheme devised by ETS in collaboration with NIE. 
It was designed to maximize compatibility with the product classification 
scheme already in use by NCERD. A complete listing of the PROP version of 
the classification scheme appears as Appendix B. A printout of the products 
in the PROP pool, tabulated according to this classification system, appears 
as Appendix C» 
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Collection and Organization of Information 
on Products in the Pool 

Once the PROP pool was stabilized at 90 products, the task of assemb- 
ling information about them that would be maximally useful to the Panel in 
arriving at dissemination recommendations began in earnest. Because the 
Panel had to evaluate and compare such a large number of products in a 
relatively short space of time, it was incumbent upon the ETS support staff 
members to produce accurate and complete descriptive information on each 
product which would lend itself to rapid familiarization and would provide 
a uniform basis for comparisons across products. 

Staff members from the ETS Test Development Division, some 26 in all, 
were assigned responsibility for gathering and organizing information about 
each product and for producing a product precis written to carefully spelled- 
out specifications. The information came to ETS from a number of sources. 
For products which had been held over from the PEP pool in 19 71, considerable 
information and, in most cases, the products themselves, were at hand. The 
product developers, when they were asked if they wanted their products 
reviewed in PROP, were encouraged to forward additional materials, partic- 
ularly any summative evaluation and validation reports which might have been 
published since 19 71, For products which came into the PROP pool as a 
result of having been recommended in the AIR Impact Study, extensive reports 
on each product were available under AIR imprimateur. Additional up-to-date 
information was sought from each product developer involved. 

For products newly nominated to the PROP pool by their developers or 
by the Bureau for the Education of the Handicapped, it proved necessary to 
gather product information de novo. The staff member assigned responsi- 
bility for preparing descriptive material on each product was encouraged 
to establish close Telephone contact with the developer as soon as that 
product's entry into the pool had been officially established. The staff 
member assigned to a particular product almost invariably had subject- 
matter expertise in the area germane to it, and in a number of instances 
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had had prior experience with it, either in a teaching context or otherv^ise. 
A number of the reviewers were, in addition, evaluation specialists. Thus 
the reviewers were particularly well equipped to interact with product 
developers and to seek out those materials and publications which would be 
most useful in preparing product descriptions for the Panel^s use. In a 
few instances, as in cases where the materials involved were exceptionally 
bulky or vastly expensive, it did not prove possible for the product 
reviewers and panelists to examine the products themselves. Tnese were the 
rare exceptions, however. 

In addition to examining the products, accompanying promotional 
literature, and technical reports on evaluation and validation studies, 
reviewers had access to information gathered in an intensive ERIC (Educa- 
tional Resources Information Center) search and in a search of the govern- 
ment's files, including the so-called C-2 reports on the state of develop- 
ment of products being funded in the various regional laboratories and 
research and development centers and entries in the CEDaR catalog. 

From the outset there was great concern for achieving uniform 
standards in product review, especially with so many different individuals 
engaged in the review process. Accordingly, at an early stage all product 
reviewers were asked to attend a meeting at which the project directors, 
project manager, and project coordinator explained the nature and purpose 
of PROP and went over a detailed document (Appendix D) which outlined the 
procedures to be followed in preparing a four-page precis on each product. 
The precis was to encompass a brief description of the product; a summary 
of available information on evaluation and validation; a discussion of 
implementation implications, including such things as special requirements 
for trained and/or additional personnel, the need for special materials 
or facilities, administrative considerations, and possible system 
disruption; costs; dissemination efforts already made by the product 
developer; possible obstacles to dissemination; and suggested dissemi- 
nation stretegies, including a projection as to possible activities which 
the availability of additional dissemination funds as a result of PROP 
might make possible. 
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A dossier was established for each product, ar d as; material was 
accumulated, or action taken, entries were scrupulously made on the dossier 
cover sheet (Appendix E) and the PROP action form (Appendix F) . In 
addition, a card file was maintained on each product, and as products were 
added at the request of developers, or dropped either bv request or as a 
result of decisions by NIE, it became possible bv a simple filing operation 
to produce at any instant a complete list of products still under consider- 
ation in the active pool. 

Bulky materials too large to include in dossier files, including such 
things as films., filmstrips, cassettes, microfiches, games, tliree- dimens ional 
models, and long reports, were stored in a convenient location and product 
reviewers were promptly advised as to their availability. Full-time 
clerical support was assigned to the project, and a log was maintained so 
that the exact location of any PROP product materials could be ascertained 
at any time. 

A valuable source of product materials was the NIE itself. In order 
to serve the needs of its own review panels, the NIE's predecessor units 
in the USOE had accumulated a large collection of educational products. 
Many of these became final entrants into the PROP pool. Since there were 
long and unforeseen delays in finalizing the PROP pool, the willingness 
of the NIE to share its product collection with ETS solved many problems 
when time was of the essence. In addition, rather heroic measures, 
including the use of volunteer couriers , were resorted to in order to 
obtain original product materials for product reviewers in time for them 
to produce thoughtfully organized precis in advance of the first Panel 
meeting. 

In order to achieve" accuracy, uniformity, and freedom from bias, precis 
preparation was monitored closely by senior staff members working on PROP. 
Each precis was revievv^ed at least t^^?ice, in most instances by two different 
individuals, and revisions were incorporated to accommodate the criticisms 
and suggestions thus generated. In a number of instances, as many as three 
revisions were undertaken. Almost all precis were revised twice. The 
precis were carefully edited, typed, proofed, and assembled into product 
O lasses in a sequence designed to facilitate Panel review. 
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On completion, a copy of each precis was sent to the product developer 
for his approval (Appendix G) . The developer was asked to check the precis 
for accuracy and to provide any supplementary or clarifying information 
available. In almost all instances the product developers concurred, at 
least in the main, with the substance of the precis. In a few instances 
minor corrections were made and some helpful evaluation or validation 
reports which had recently become available were supplied as a result of 
this step. Almost all product developers had an opportunity to react to 
the precis on their products in advance of the first Panel meeting. In 
those instances where time pressures did not permit chis, there was ample 
time to permit developers* comments to be brought to the attention of 
panelists by means of supplements prepared for and distributed at the 
second Panel meeting. 

Before the Panel meetings, the precis were organized into loose-leaf 
subpanel books, by NCERD specialty area, and given %ook numbers" for 
convenience in location, reference, and retrieval. The precis books 
served as the primary source of information for the panelists in carr3?'ing 
out their comparative review* Panelists were, of course, encouraged to 
consult product dossiers and the other original materials available rather 
than to rely exclusively on the information in the secondary sources. To 
assist in this process, such materials were made readily available at both 
Panel meetings, catalogued and located for easy reference, and ETS staff 
members provided assistance in locating desired materials. A variety of 
equipment, including slide and film projectors and cassette players, was 
available at Panel meetings so that multi-media materials could be 
observed directly. 

During the interval between the two Panel meetings, product reviewers 
provided additional information sought by panelists, either via a 
reanalysis of available dossier materials or by requesting additional 
information from product developers. In one instance a site visit was 
made by an ETS staff member to seek the information desired and in two 
other instances, where the focus was on test reviews, measurement experts 
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in ETS's Developmental Research Division made extensive independent and 
comparative judgments as to the strengths and weaknesses of the tests 
involved. All these materials were available to panelists at the start 
of their second meeting in February. 

After the Panel had made its final recommendations at its second 
meeting, product reviewers provided expository discussions, aimed at a 
lay readership, of all products either given dissemination recommendations 
or suggested for review in the future. The resulting product descriptions 
form the body of two auxiliary documents. That for products given dissemi 
nation recommendations accompanies this report » 



Development and Refinement 
of Selection Criteria 

Introduction . In the 1971 project for the selection of products for focused 
dissemination, a major concern of panel members was the formulation of a 
detailed set of criteria for evaluating products. As the panel^s conception 
of criteria evolved through discussion and through application of criteria 
to products, Michael Scriven developed a series of worksheets, each reflect- 
ing the current status of the panel's thinking. The worksheet shown on 
page 27 is two generations beyond the eighth version of the 1971 worksheet 
prepared by Scriven after the completion of the 19 71 selection process. It 
was designed after the ninth version was subjected to review by the 1972-1973 
Panel and by the NIE in the months prior to the Panel's first meeting. 

The evaluation strategy invoked through use of the worksheet focuses on 
four broad characteristics of each product: (1) Goals, (2) Effectiveness, 
(3) Costs, and (4) Adoptability. Within each of these areas a number of 
detailed considerations are identified as having a bearing on the 
evaluation of that product characteristic. Provision is made for one or 
more ratings for each characteristic. In arriving at an overall evaluation 
of the product, each evaluator used the information available on the 
product and the detailed judgments recorded on the worksheet in whatever 
way seemed most appropriate » Neither formal nor informal weightings for 
the various aspects of the criterion were called for* Thus the detailed 
analysis of the criterion was intended to aid in making judgments and to 
facilitate discussion of products rather than to impose any formal 
relationship between the overall evaluation and the evaluations made 
earlier on a more detailed level of analysis . 

Criteria for Selection of Products . The entries on the Product Evalu- 
ation Worksheet itself are necessarily brief. Understanding of it may be 
enhanced by studying the amplified statements on pages 26-34 as the form 
is reviewed* 
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Goals > The extent to which the goals envisaged by a product developer 
are significant for education. Five specific aspects of this criterion 
may be identified: 

(1) Urgent present need — Does the product address itself to urgent 
needs? 

(2) Educational central! ty — Does the product concern itself with 
outcomes which are central to the educational process rather than 
with special, limited outcomes? 

(3) Size of target population — How large is the group for which the 
designer considers the product appropriate? 

(4) Notable originality — Does the product embody well-conceived 
innovations in content, method, or both? Does it reflect old 
orientations or new ones? 

(5) "Leverage^' or ''multiplier '' — Would use of the product be likely 
to facilitate the adoption of other educational improvements by 
the school? 

Evaluation of these specific considerations is to be expressed in terms 
of three options: 

+ Significant positive factor 

- Significant deficiency or drawback 

? Unable to make a reliable judgment on the available evidence 
This coding scheme for the evaluation of specific considerations is used 
throughout the worksheet except in the case of Adoptability . 

The overall rating for Goals is to be expressed on the following scale: 

A Excellent 

B Good 

C Fair 

D Poor 

F Undesirable 

T .Insufficient Data 

A relatively hig?i degree of confidence in the rating made is to be expressed 
by a double circle. 
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Kvjlualion of Conskicrarions 



Judgment ('odes < Sign if icu til positive 
faciors or deficiencies) 
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Significant deClcieney or drawbuek 
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available evidence 
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Si/.e of target population 
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Sample fairness 
Adequacy of controls 
Validity of criterion 
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Ef f ectiveness > Hie extent to which the product may be expected to achieve 
its stated goals with the target population and the extent to which it may 
be expected to produce desirable or undesirable side effects. Side effects 
may include outcomes other than those included in the stated goals or effects 
on populations other than the target population and may be either positive or 
negative. 

In evaluating effectiveness, a major group of considerations is con- 
cerned with the adequacy of test data , as follows: 

SamgjLe size — Were studies conducted on a substantial body of students 
in a number of classrooms? 

Sample fairness — Were the study samples reasonably representative of 
the target population? 

Adequacy of controls — Did the experimental design include suitable, 
well-defined comparison groups together with the treatment group? 

Validity of the criterion — Were the measures of educational outcomes 
appropriate to the stated goals for the product? 

Objectivity of judgment — If ratings by administrators, teachers, or 
students were obtained, did the ratings focus on relatively concrete, 
observable potential outcomes? 

Evidence of long-range effects — Were follow-up studies made to 
determine whether the effect*^ observed at the end of the experiment were 
sustained over a substantial pexiou of time? 

Another group of considerations is concerned with the actual performance 
of the product in field tests , as follows : 

Student Performance Changes — Is there evidence that changes in 
student performance occurred under realistic conditions? 

Student Attitude Changes — Is there evidence that students who 
participated in field trials responded favorably (or unfavorably) to the 
product ? 

Teacher Reactions — Is there as to how teachers reacted to various 
aspects of the product? 

Two further considerations are concerned with possible supplementary 
evidence, as follows: 
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Background Evidence — Is there reason to believe that the earlier 
success of the developer in related fields is likely to contribute to the 
effectiveness of tlie product under consideration? 

Internal Evidence — Is there evidence of care and systematic effort 
In the development of the product which supports ^or fails to support) the 
empirical evidence as to effectiveness? 

The worksheet provides space for describing any significant side 
effects for which evidence is available. 

The performance Rating on Effectiveness of the product in achieving 
its stated goalo is to be expressed in terms of the same rating scale as 
that used for Goals » as follows: 

A Clearly Effective 

B Good 

C Fair 

D Poor 

F Inadequate 

? Insufficient Data 

Some problem exists in expressing judgments as ^J side effects because 
the unintended outcomes of a product may be either desirable or undesirable. 
The present worksheet provides for separate ratings of the extent of side 
effects oii the scale used for performance rating and the desirabili ty of 
side effects on the scale which is used for Goals , as shown on page 26, 
For side effects which are desirable, the meaning of the two scales should 
be clear. A performance Rating on Side Effects would indiccite the extent 
to which clear evidence of the operation of the side effect is available 
and the rating on Desirabili ty of Side Effects would reflect how desirable 
the outcome is judged to be. Thus, if a side effect is undesirable but 
there is strong evidence that it exists, the Rating on Sjde Effects might 
be recorded as "Excellent" to indicate that clear evidence of the ef f 
was available and the rating on Desirability of Side Effects would be 
recorded as "Undesirable." On the whole, it seems better to think of the 
Rating on Side Effects as an evaluation of evidence that the product is 
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in fact producing the designated side effects and to allow judgment as to 
desirability to affect only the rating designated Desirability of Side 
Effects . 

Cos ts : The extent to which the introduction and subsequent use of a product 
place heavy demands on a school's resources, both economic and human. In 
developing detailed criteria related to costs, the 1971 panel noted that the 
introduction of a product might result in reduced costs, part: cularly operat- 
ing cos ts J in some instances. 

Three considerations related to Costs of Materials may be identified, 
as follows ; 

Basic materials: initial year 

Basic materials: continuing cost per year 

Enrichment materials 

A number of other costs inherent in the introduction of a product 
deserve consideration, as follows: 

In-service training costs 

Repair, upkeep, running, and temporary replacement costs . 
This consideration includes costs which could reasonably be anticipated to 
keep the product in use over and above the basic continuing operating costs. 

Extra (or less) space needed 

Special needs (e.g., projection room ) 

Consultant cos ts 

An important group of considerations is concerned with whether additional 
(or fewer) staff members of various kinds would be needed if the product were 
put into use. Separate evaluations are to be made fo.r four kinds oi' staff 
members, as follows: 

Professional 

Paraprofessional 

Technical 

Clerical 
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In addition to the dollar and time costs of a product, there may be 
attitudinal factors which deserve consideration in assessing the total 
"cost" of an innovation* It is plausible, of course, that attitudinal 
factors may be either unfavorable or favorable. Accordingly, the work- 
sheet provides for judging the opposition to (or support for) the product 
on the part of the : 

" .... '. 
Student . . 

Staff 

Communi ty t ■ - ^^i. 

Finally, consideration may be given to the fact that certain- products 
tend to disrupt the operation of the system as a whole. Provision is made 
on the worksheet for noting that installation of a product may be "system- 
disruptive." 

For the overall evaluation of Cos ts ^ separate rating scales are pro- 
vided for Ins tallation Cos ts and Continuing Cos ts , as follows : 
Installation Costs Continuing Costs 

Maj or Major 
Substantial ^ Substantial 

Modest Modest 

Potential Saving 

It must be acknowledged that evaluation of a , product with respect to costs 
is very difficult without specific information about the school system in 
which it is to be used and without information as to the cost of alternative 
ways of achieving the same goal. Consequently, the categories for rating 
costs are relatively broad. and do not, except for the "potential saving" 
category under Continuing . Cos ts , call for a comparative evaluation. 

Adoptability : The extent to which the product is readily available to a 
school which wishes to adopt it. 

The detailed considerations for Adoptability involve primarily descrip- 
tion of current status rather than evaluation. Consequently, judgments are 
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expressed by a check mark rather than by the +, ? code used for other 
considerations. Concerning Materials currently available from the author 
or publisher, the worksheet provides for checking any of the following 
' ' that apply : 

Sample only • 
Limited quantities now 
Unlimited quantities now 

Tests and other supplementary materials also available 
Enrichment materials 
Situation unclear 

Another aspect of Adopt abili ty concerns preparatory steps to be under- 
taken by a school system before a product can be put into use. Under Use , 
the worksheet provides for checking any of the following that' apply: 

Requires advance planning 

Requires plant changes 

Requires support system contracts 

Evaluation of Adopt abili ty is concerned with the current s tatus of the 
product. Four options are provided, as follows: 

Could readily be installed with modest planning effort 

Could be installed with substantial planning effort 

Not ready for widespread use - 

Status unclear 

Overall Evaluation ; Presently demonstrated desirability for inclusion in 
the pool- of products recoiranended for dissemination. 

Ihe four major characteristics in the evaluation strategy are clearly 
relevant to an overall judgment. In addition, the following two consider- 
ations ma)^,' at least for some products, have an important bearing on the 
judgment: 

Whether or not other dissemination support is available 
Whether or not at least one competing product is available 

O 
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A Panel member's Overall Evaluation of a product is to be expressed 
in terms of a personal judgment as to the desirability of including the 
product in thiB recommended product pool. The six options are: 

5 Definitely should be included 

4 Good prospect 

3 ' Will accept 

2 Bad prospect 

1 Definitely should not be included 
H Hold for next year's pool 

Finally, the worksheet provides a box which may be checked to indicate that 
comnients appear on the back of the worksheet. Such comments constitute an 
important aspect of the entire evaluation process. In particular, comments 
may include: ■ 

i 

(a) Conditions under which a designated higher Overall 
Evaluation rating would be given 

(b) Statement of the main factors leading to the favorable 
or unfavorable rating given to a product 

(c) Specification of missing information about a product 
which the developer should have supplied or the analyst 
should have secured 

Refinement of Criteria , As noted in the introduction to this section, the 
ninth version of an Evaluation Worksheet was reviewed by members of the 
1972-19 73 Panel in preparation for the design of a worksheet for PROP be- 
fore the first Panel meeting. On the basis of the comments returned by 
panelists and further study by senior staff members working on PROP, a 
tenth version of the Evaluation Worksheet was designed. That revision is 
the form shown on page 27. It was this tenth generation in the evolution 
of PEP- PROP Evaluation Worksheets that was used in PROP 19 72-19 73. An 
amplified statement similar to that given here accompanied the form to 
refresh the memories of returning panelists .and to orient new members as 
to the current state of the criterion statement. 
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Panel Procedures in the 
Initial Stages of Product Review 

Introduction . The initial PROP pool of products was almost twice the size 
of that of PEP in 1971, and two Panel review meetings, rather than the 
three held in 1971, were contemplated for PROP, A further complication 
came about as a result of the unexpected slippage of time in resolving 
the question of how many and what kinds of products would go into the 
initial pool for Panel review. It was therefore necessary to plan the 
activities of the Panel with great care, to ask Panel members to complete 
certain tasks before and between the two scheduled meetings, and to expect 
panelists to put in long and intensive sessions during both meetings and 
to keep up-to-date on their individual review of product's by long hours of 
work outside of meetings. It was also necessary to schedule the Panel 
meetings as late in the project calendar as feasible to allow adequate 
time for almost all staff product analyses to be prepared in advance of 
the first amd all to be completed well before the second meeting. The two 
starting dates were a month apart, the first meeting being held January 
10-12; the second, February 7-9, 19 73, 

As preparatory steps during late November and early December, panel- 
ists reviewed and were asked to comment upon a suggested set of Panel 
procedures and, as noted earlier, to make modifications in the Evaluation 
Worksheet which served to put in shorthand version the criterion base upon 
which Panel consensus was essential before definitive judgments could be 
made in the product review process. As further preparation during late 
December and early January, the panelists familiarized themselves 
individually with the analytical precis on six products sampled from the 
initial pool to test their ideas on the criterion base, to see if the 
criterion base would satisfy their own standards in dealing with these 
test cases in comparative product evaluation, and to take the beginning 
steps in standardizing the product evaluation process for PROP 1972-1973, 

Preparations for Panel Review , It would seem to be instructive first to 
look at the plans that were made for the Panel's work and then to review 
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th e events that actually occurred. To this end, the planning memorandum 
sent as a draft to PROP Panel members is shoi^rn in full on the following 
pages . 

Initial Stage of Product Nomination . Among the early decisions made once 
the Panel's first session was under way were acceptance of the Evaluation 
Worksheet dated January 1973 as the tool to guide panelists in applying 
the accepted criteria to their review of individual products (see page 27) ; 
concurrence in the organizational pattern calling for two subpanels, each 
with responsibility for the initial screening of about half the 90 products 
scheduled for review; and agreement to start out in each subpanel with 
group discussion of as many as necessary of those products for which precis 
had been reviewed before arrival, until each panelist was confortable in 
proceeding further with individual product review on his or her own. After 
the panelists worked through two or three cases as a subpanel, subsequent 
study of product- related information during the remaining sessions of the 
first meeting and the first half of the second meeting was done individually 
or in two-person teams. 

The typical pattern was for the five members of a given subpanel to 
convene periodically to reach decisive action on a block of four to six 
products each had finished reviewing and then to return to information 
sources to prepare for the next session together. During one of these 
subpanel discussions each group came to a decision that the process would 
have to be accelerated if the task were to be accomplished on schedule. 
A similar modification was installed in both subpanels. A two-person 
team in each group, consisting of those most experienced in systematic 
materials evaluation, specialized in the validation-verif ication- 
effectiveness aspects of product review while the remaining three panelists 
concentrated on factors related to goals, adoptability, and costs. The 
results of giving a focus to each panelist's search seemed to be that 
products were given more thorough review in the time available and that 
panelists had fuller understandings and insights related to each product 
at the time decisive subpanel action on it was taken than would otherwise 
have been the case. 
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EDUCATION AI^ TESTING SERX'ICE 



PRINCE:^rON, N.J. 085 lO 



Area Codf 609 
921-9000 
CABLE'ED UCTESTSVC 

Panel Review of Products (PROP) ^ ^ A Z Z 

Tentative Outline of Panel Activities November 21, 19 72 

Introduction , The identification of exemplary, research-based products 
which hold promise of effective educational use is the essential goal of 
the Panel's activities. Within the limitations imposed by the number of 
products (100 plus) to be judged and the time available for completing 
the judgments, the Panel has full responsibility for determining criteria 
and procedures to be used in evaluating products, for devising procedures 
which will insure that panel decisions fairly represent the judgments of 
Panel members, and for establishing its own" work schedules. This paper has 
been prepared to serve as a basis for review and reaction by individual 
panelists prior to the first meeting. Completion of these steps at this 
time will bring the tasks ahead into feasible bounds. 

Outline of Activities 

One possible sequence of steps in the evaluation process may be 
described as follows : 

a. Formulation of criteria . This step is concerned with reaching 
a consensus with respect to the characteristics of a product 
and the kinds of evidence about the product which need to be 
considered in evaluating it. To the extent that Panel members 
can agree about the kinds of information that should be con- 
sidered, the final decision process should be facilitated. A 
proposed Product Evaluation Worksheet, based primarily upon the 
criteria developed in the 19 71 Product Evaluation Program, is 
attached. A criterion statement will be drafted when comments 
from panelists are in hand. 

b . Reading of product summaries (precis) and examination of basic 
source materials (dossiers) by Panel members . It is clear that 
a substantial portion of the Panel's time must be allocated to 
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Tentative Outline of Panel Activities November 21, 1972 

familiarization with each product to be evaluated. For this 
purpose, analytical summaries (precis) prepared by ETS staff 
members will be available for each product. In addition, 
one or more folders of basic source materials (a product 
dossier) will be available for each product. 

c. Judging the quality of a product . This step will presumably 
involve an independent judgment based on a consideration of 
relevant information by each Panel member. 

d. Arriving at a decision concerning the product and documenting 
the decision . This step requires a procedure for making Panel 
decisions which takes full account of the views of each member. 
Further expression of the Panel's evaluation of specific 
aspects of the product would be desirable. 

Proposed Two-Stage Selection Process 

It is recommended that for purposes of making an initial selection 
the Panel be divided into two five-member subpanels. Each subpanel would 
be responsible for performing the initial selection on approximately half 
of the products in the 1972 product pool. In assigning products to sub- 
panels, care would be taken to assign products designed to perform similar 
educational functions to the same subpanel. Each subpanel would be asked 
to select 15 to 25 products for consideration by the entire Panel in the 
final selection stage. 

If the two-stage selection plan is used, and if half or fewer of the 
products enter the second stage, the amount of time devoted to evaluating 
the products would be about one-fourth less than would be required if every 
Panel member were to evaluate every product. The time thereby saved could 
be devoted to a more intensive scrutiny of the products finally selected 
from the reduced product pool. 
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Tentative Outline of Panel Activities November 21, 19 72 

Proposed Schedule of Panel Activities 

a. Before the f irs t meeting . As suggested in the covering letter 
for this memorandum, Panel members are asked to comment in 
writing on the proposed work plan and on the attached 
Evaluation Worksheet. In addition. Panel members will be asked 
to familiarize themselves with the precis for six products. 
These six precis will be mailed to Panel members before the 
first meeting. 

b. During the first meeting . It is recommended that the first 
main agenda item be a discussion of criteria and the Evaluation 
Worksheet so that revisions of the worksheet can be made before 
evaluation of products begins. The second main agenda item 
would then be the formulation of working procedures and a 
tentative schedule of Panel activities. If the Panel decides 
that the evaluation of products should be done during the Panel 
meetings, when members will be free of interruptions and will 
be able to refer to product dossiers, it is clear that a large 
fraction of the two-day meeting will be needed for this kind of 
activity. If the Panel decides to operate during the initial 
screening stage as two subpanels, each fully responsible for 
the approximately 50 products assigned to it, each subpanel 
might begin work by evaluating the six products mailed in 
advance, followed by a full Panel discussion of the six products. 
After this general session, however, most of the remaining time 
could be devoted to evaluation of products by Panel members, to 
discussion of ratings, and to reaching a tentative decision 
concerning the products to be considered in the final selection 
stage. 

c. Between meetings . If the delegation of responsibility to sub- 
panels is adopted, and if final decisions on inclusion of 
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Tentative Outline of Panel Activities Noveinber 21, 1972 

products in the final selection stage are made at the secuad 
meeting, it would be desirable if Panel members refreshed their 
memories concerning products before the second meeting. In 
addition, it is possible that precis for one or more products 
. , may not be ready for the first meeting. If so, it would be 

desirable for Panel members to familiarize themselves with 
these precis before the second meeting. Some Panel members 
might wish, also, to familiarize themselves with the precis 
of all products considered by the other subpanel (or of those 
products tentatively selected for the final selection stage), 
d. Second meeting . The second meeting would begin with an 

opportunity for reference to product dossiers and for evalua- 
ting any products for which a precis was not ready for the 
first meeting. Each subpanel would then complete its decisions 
concerning products to be included in the final selection stage. 
Each subpanel member would then evaluate the products which 
were selected by the other subpanel. As far as possible, the 
second day of this meeting would be devoted to discussions of 
products, to selection of not more than 35 products to be 
included in the "Recommended Product Pool/' and to documentation 
of the decisions reached. As part of the documentation. Panel 
members may wish to make final revisions of their Evaluation 
Worksheets, including comments on the products. These final 
worksheets would then be summarized by the staff after the 
meeting for inclusion in the final report of the Panel^s work. 
In these summaries, there would be no attribution of ratings 
or comments to particular Panel members. The summary report 
would be mailed to all Panel members for review and comment 
before it is forwarded to the National Institute of Education, 
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Tentative Outlina of Panel Activities November 21, 19 72 

Proposed Mechanism for Subpanel and Panel Decisions 

It is suggested that each Panel member make an overall judgment on 
the suitability of a product for inclusion in the Recommended Product 
Pool and express his evaluation on the following scale: 

5 Definitely should be included 

4 Good prospect 

3 Will accept 

2 Poor prospect 

1 Definitely should not be included. 

At the completion of the initial reading session, product latings by each 
subpanel member would be considered by the entire subpanel. Products would 
then be discussed and each subpanel member would be asked to rerate each 
product in light of the discussion. If the suggested scale is adopted, it 
would appear that the irinimum total rating for acceptance would probably 
lie between 15 and 20 in the initial selection stage. A subpanel might 
wish to discuss further any products which were just above or just below 
the point selected as the minimum* 

Each product, at the final selection stage, would have been evaluated 
by all Panel members. At this stage, the rating of a product by each Panel 
member and the significant characteristics of the producr would be 
discussed by the entire Panel. This would be followed by a rarating of 
each product. The Panel would need to establish a minimum acceptable total 
rating and might wish to make a final review of all products near the 
minimum acceptable total in order to insure that each product selected is 
an exemplary, research-based product which holds promise of effective 
educational use. 
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The mechanics of record-keeping on subpanel activities utilized the 
Evaluation Worksheet. Each panelist prepared a worksheet for each product 
assigned to his or her subpanel, and all members proceeded to move through 
the products in his book by book order number. Each member had his o\vn 
"out-box" which was monitored by the staff. Worksheets were periodically 
collected, recorded, and returned to each panelist's "in-box." As a 
consequence, when a subpanel session turned to a given product, reference 
to the record would provide a summary of overall evaluation ratings on 
that product, and the discussion could quickly zero in on pattens of 
convergence and divergence in Panel judgments, together with their possible 
implications . 

The "Blue" subpanel, consisting of Blanchard, Bollenbacher , Gaines, 
Komoski, and Scriven, assumed responsibility for initial review of 47 
products. These were classified in three NCERD Specialty areas as follows: 
B. Planning, management and evaluation systems - 15, C. Materials for use 
in instructional personnel development - 15, and F. Materials for home- 
school intervention in the early years - 17. The results are summarized 
in the table shown below. 



BLUE SUBPANEL INITIAL NOMINATIONS 


Product Classification 


B 


C 


F 


Total 


Number of Products 


15 


15 


17 


47 


Nominated for Plenary 
Panel Review 


7 


1 


3 


11 


Nominated as Hold for 
Another Year 


3 


5 


2 


10 


More Discussion and/or Data Needed 


4 


5 


6 


15 


Probably Not for Nomination 


1 


4 


6 


11 



-43- 



The "Green*' subpanel, consisting of Krathwohl, Lumsdaine, Mancuso, 
Trollman, and Watson, assumed responsibility for initial review of 43 
products. These were classified in three NCERD Specialty areas zs follows: 
A. Materials for use in theory and knowledge building for organizational 
change - 9, D. CvLricular programs - ?9 , and E, Culturally- targeted 
curricular programs - 5. The results are summarized in the following table. 



GREEN SUBPANEL INITIAL NOMINATIONS 


Product Classif icatior? 


A 


D 


E 


Total 


Number of Products 


9 


29 


5 


43 


Nominated for Plenary 
Panel Review 


0 


6 


0 


6 


Nominated as Hold for 
Another Year 


1 


7 




10 


More Discussion and/or 
Data Needed 


1 






18 


Probably Not for Nomination 


7 


2 


0 


9 



*Some products in these categories were not subjected to review during the 
initial stage of product nomination at the first meeting* 



Final Stage of Product Nomination , Until 9 a.m. February 9, the hour 
scheduled for a plenary session, the second meeting was a continu. ' ion of 
subpanel deliberations. Both the January and the February meetings ran well 
into the night. The agendas were extremely full. Precis on three products 
were not available until the second meeting. An additional eight products 
had not had the benefit of subpanel discussion, and at least 24 others had 
been slated for extended subpanel considerations by the end of the first 
meeting. On a number of products in this latter group, additional informa- 
tion had been secured from developers between the meetings, and in one case 
a site visit had been made by an ETS professional staff member. Finally, 
each subpanel took on the added task of reviewing the other subpanel's work 
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and entered demurrers in those instances where the overall judgment differed 
as to the course of action recommended. 

The Blue subpanel concluded its work with nomination from its pool of 
17 products as candidates for dissemination recommendations of some kind. 
As will be seen in the next section, not all nominations survived the 
plenary session. In a separate action, a limited number of products were 
designated to be held over for review in another year. 



The Green subpanel nominated from its pool eight products as candidates 
for some sort of dissemination recommendation; three additional products 
from its pool, making 11 in all, were nominated for dissemination-oriented 
action as a result of the plenary discussion. Here again, not all nominated 
products survived the plenary session. The Green subpanel also identified 
products to be held over for review in another year. Its list in that 
category was considerably larger than that of the Blue subpanel. 



As it turned out, the total numbers of products on which the two sub- 
panels made recommendations for subsequent action related either to 
disseminaticn-related matters or later comparative review were about equal. 
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Panel Procedures in the Final 
Stages of Product Selection 

Introduction . There were three parts to the panelists* preparations for 
their plenary product selection session. The two series of sub panel 
sessions, the first the month before and the other the day before, had 
given each panel member an opportunity to become fully acquainted with 
all the products likely to be nominated for full-Panel review. Each 
panelist had his own set of product precis, on both his subpanel's 
products and on those of the other subpanel. Dossiers containing full 
documentation on all products had bee'^ readily available and were heavily 
used. Updated supplements to product precis had also been issued at the 
start of the February meeting and were posted to panelists* product precis 
books. The third element of preparation was that the Panel members had 
had their precis books with them between the January and February meetings, 
together with analyses supplied by staff members summarizing the results 
of *heir deliberations in January. By reviewing the tabulations supplied, 
each panelist could ascertain with regard to products in his subpanel 's 
pool and in the pool of the counterpart group the overall evaluation of 
each panelist (by code rather than by name) on each product. He could 
also tell in the case of an individual product whether, according to 
initial subpanel judgment, a product seemed to be headed toward a dissemi- 
nation recommendation, a hold category, or some other classification. 

Panel members were encouraged to devote attention between the meetings 
to reading precis from the pool of their counterpart subpanel, but to 
concentrate on those products which had received ratings high enough in 
the initial stage to be possible candidates for the final pool, leaving 
aside those which the other subpanel had rated low or otherwise indicated 
should be rejected. On the other hand, they were also urged to take the 
opportunity to put up for reconsideration any rejection-bound product they 
might identify as having special appeal to them. 



As a consequence of the preparations made, the Panel members had 
acquired an extensive background, had made reminder notes on their Evalua- 
tion Worksheets, and had organized their own worksheets for easy reference 
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when a given product was brought up for plenary review. In most cases, 
all panelists had worksheets on every product and could enter into the 
discussion and decision on each product as it was nominated for plenary 
consideration. • In the few cases for which this was not true, panelists 
with the product well in mind gave a resume' to refresh memories so that 
both discussion "and decision could in fact be representative of the Panel 
as a whole. 

Panel Agreements on Decision Categories . It had become apparent at the 
January meeting that the Panel members were not inclined toward a plan 
for plenary session decisions which would lead to clear-cut selection of 
certain products and to certain rejection of others < The collective 
judgment seemed to be that there were so few clearly outstanding, convinc- 
ingly verified, and demonstrably cost-effective products among those 
reviewed as to make for "no competition." Certainly it would not have 
been feasible for a final pool of products as large as the maximum 
contemplated in the original plans of the NIE (35) to have been designated 
for out-and-out dissemination action. 

The thinking of Panel members had turned early to the possibility 
of indicating in the case of certain products that seemed promising a 
variety of levels of dissemination for which sound justification might 
be reasoned and of indicating with respect to a given product specific 
reservations which the NIE might take into account in the process of mov- 
ing ahead with dissemination plans. In moving toward such a decision- 
making model, the Panel cleared the way for admission into the final pool 
products whose . likely future promise could be perceived but whose current 
viability could not be established. 

In advance of the February meeting, panelists were sent a proposed 
set of decision categories as shown on the following page. The choices 
suggested represented an expansion of the overall ratings in the lower 
right-hand comer of the Evaluation Worksheet,- with approximate relation- 
ships as shown in the next chart. 
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Proposed 
19 73 DECISION CATEGORIES 
PANEL REVIEW OF PRODUCTS 



PROP AC PROP BOOK No. 



Descriptors 



Rater Name/Code Date 



I I A. This product is exceptionally deserving of dissemination . 



B. This product deserves to be disseminated. 



C. This product deserves to be disseminated subject to the following 
reservation(s) : 



I I D. This product deserves limited dissemination solely for the purpose 
of obtaining field-test data on effectiveness. 



] E. This product would deserve to be disseminated if the following 
deficiency (cr deficiencies) were removed: 



] F. This product should not be considered for dissemination until 

substantial additional evidence of its effectiveness in achieving 
its stated goals has been reported. 



] G. Further research or development is needed before an adequate 
evaluation of this product can be made. 



I 1 H. This product should not be disseminated for the following reason(s): 



ETS Educational Testing Service, Princeton, New Jersey January 19 73* 
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Overall Ratings On 
Evaluation Worksheet 


Decision Categories 
Proposed 


5. Definitely should 
be included- 


A. Exceptionally deserving 
of dissemination 

B , This product deserves 
to be disseminated 


4. Good prospect 


C. Deserves to be disseminated 
subject to reservation 


3« Will accept 


D« Deserves dissemination to 
obtain more field-test data 

E, Deserves dissemination if 
deficiency is removed 


2, Bad prospect 


F, Should not be considered without 
more evidence on effectiveness 

G, Requires more research and 
development prior to evaluation 


1. Definitely should 
not be included 


R, Should not be 
disseminated 



In this decision model, products which were judged as appropriate to 
be held for the future or as rejections would have a majority (6 or more) 
of the panelists giving ratings of F, G, or Where majority ratings were 
higher, the Panel would be guided by the mix in its voting pattern in the 
process of framing a product-explicit recommendation. 

With both Panel and staff preparations as background, the Panel turned 
to discussion as to the form its decisions would take. The Panel was not 
particularly enamored of the decision categories the staff had proposed, 
although it did persistently pursue the objective of gaining considerably 
more latitude than a ''go-no go" (in or out) focus would have permitted, 
A substantial block of time was spent at the outset, therefore, in set- 



id 
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tling upon a method that would enable effective and useful decisions to 
be drawn by way of Panel discussion of the overall judgments — both 
quantitative and qualitative — that had resulted from earlier individual 
study by panelists and from discussion in subpanel sessions. What the 
Panel sought, essentially, was an adequate basis for communicating a broad 
array of recommendations to the NIE concerning the more promising products 
among those that had been brought to its attention. 

The reasoning that evolved from the Panel's coping with the categori- 
zation problem ran somewhat as follows: 

o Most products entered into the initial 1972-1973 PROP product 
pool seemed '*betwixt and between.** They were not far enough 
along to be vigorously disseminated (such as through commerical 
distributors) but were far enough along to appear to developers 
and to a lesser extent to panelists to be promising (even though 
hard data were often not yet in). Under such circumstances, it 
seemed highly unlikely that any substantial number of products 
in the initial pool would be recommended as strong prospects 
for dissemination funding or, for that matter, recommended for 
dissemination funding conditional upon minor refinements and 
improvements. 

0 The PROP Panel saw the products and their potential from the 
vantage point of a disinterested neutral corner. Its collective 
judgment, it would seem, could be applied beyond selection of 
products for NIE dissemination to include suggested alternative 
courses of action that might be indicated in order to justify 
that vigorous dissemination actions be taken--ei ther currently 
or later. Gaps in a product's development might be identified, 

^ or the need for additional field- test data, or the need for more 

1 or different product evaluation activities. 

o Perhaps the PROP Panel should extend the dissemination-oriented 
recommendations it makes for the NIE to take into account such 
gaps as those that have been identified during product review. 
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and to propose the relationship that would seem appropriate be- 
tween the closing of these gaps and dissemination funding for a 
given product. 

The approach taken by the Panel to easing the stringency of selection 
requirements seemed sound to the observers present from NIE. The Panel, 
under Krathwohl^s guidance, consequently proceeded to define six differienti- 
ating categories into which to channel decisions as to dissemination 
recommendations : 

1. An ideal product, well evaluated and ready to go (a rare occurrence). 
Here, dissemination funds could be recommended without reservation. 

2. A promising product about whose effectiveness the developer must learn 
more and for which he needs money. Here, a combination of field trials and 
dissemination might be recommended as concurrent actions, even with 
attendant risks. 

3. A promising product for which money is needed for independent review 
of available data or for the execution of an independent evaluation. Here, 
a recommendation would probably call for completion of these steps. prior to 
dissemination fundi,ng, although in some cases concurrent funding of the 
additional evaluation work and dissemination activities might be indicated. 

4. A promising product that needs to be carried farther along specified 
lines, such as further development, more extensive field trials, or other 
actions ordinarily considered part. of predissemination research and 
development, either by the original developers or by others who would take 
over, carrying the product's development to completion. Here, the 
recommendation would probably be limited to matters related to completion 
of the research-development-evaluation cycle, leaving it to a subsequent 
panel to draw conclusions regarding dissemination. 

5. A promising product, already commercially distributed, for which a 
dissemination subsidy is judged as probably the difference between "making 
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it or not." Here, the dis tinction would not be the kind of dissemination- 
oriented recommendation made, but rather the kind of product to which the 
recommendation would apply, 

6, A promising product, already commercially distributed, the only one 
of its kind currently available, for which there is strong evidence of its 
effectiveness in producing improvements in education. Here, a recommendation 
would reflect an assumption that one of the prime objectives of the 
dissemination activities of the NIE is getting the best available practices 
(products) into wider use, whatever their source. 

It should be emphasized that in delineating the six categories for 
differentiating among promising educational products, the Panel's frame 
of reference was somewhat broader than the original mission of PROP 1972- 
19 73 and it envisaged a broader field than the product pool it had 
subjected to comparative review. It also took account of promising 
products, commercially distributed^ which were in competition with some of 
the products in the PROP pool. The Panel reasoned that justification could 
bie made for applying the dissemination efforts of the NIE to the diffusion 
of a commercially available product as well as to that of a product coming 
out of federally funded research and development that had not yet attracted 
commercial support for distribution. Two possible bases for justification 
would be the demonstration that the commercial product was superior or 
that it was good but on the threshold of failure in the marketplace. 

Having clarified its decision-making process to the extent described, 
it remained for the Panel to settle upon the balloting procedure that it 
would use in recording its distinctions in the form of dissemination 
recommendations. The decision was to stay with a five-choice ballot, 
where each point has its own meaning but is not necessarily related to the 
other points, as on a scale. The points and their ascribed meanings, to- 
gether with approximate equivalents in the six differentiating categories 
detailed above, follow. 
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Decision Points for 
Panel Balloting 


Di f ferentiating Categories 
for Nominated Products 


A. Definitelv 
Recornmended 
(Dollar Support) 


1 An iHpfll nfoduct 

5. A promising product in 
trouble in the market place 

6. A promising commercial 
product known to be effective 


B . Condi ti onally 
Recommended 
(Dollar Support) 


2. A promising product needing 
more verification support 


C. Promising But 
Not Yet Ready 
(No Dollar Support) 


Hold. A product with some promise 
but not ready for a 
dissemination recommendation 


D. Action Prior To 
Dissemination 
(Dollar Support) 


3. A promising product needing 
independent evaluation 

4. A promising product needing 
further development 


F. Definitely Not 
Recommended 
(No Dollar Support) 


Drop. A product whose promise is 
not apparent 



The mechanics for completion of balloting called for considering 
initially the products nominated by each of the two subpanels for full Panel 
review and then for considering other products nominated by individual panel- 
ists which had some special appeal to them. As each product was called for 
consideration by the plenary session chairman, a spokesman from the nomina- 
ting subpanel or the individual "sponsor" would give a product resume"^ citing 
positive and negative factors briefly, and Panel discussion would flow from 
that. Once the bases "^f judgment had been fully aired, the panelists voted 
by a show of hands. 



No attempt was made to gain consensus, but if further discussion 
in the process of attaching caveats seemed to indicate a central tendency. 



convergence was tested. After the conditions and reservations had been 
clearly stated and the final votes decided for each product, the ballot 
and its dissemination recommendations were put into the record. 



\ 



Sunimary of Panel Selections 
and Product-Related Actions 

Introduction . As noted earlier, the Blue subpanel nominated 17 products from 
its group as candidates for dissemination-oriented recommendations. One was 
voted by eight panelists as a "C" (hold for later review) for lack of evidence 
of success and by two panelists as "F" (judged to be a poor prospect). This 
product was put into the "hold" category. A second product was voted by nine 
panelists as an "F" (not recommended), with one panelist abstaining. There 
was no clear-cut evidence as to verification, no futi^re plans for ^rield test- 
ing were discernible, and the product was kncTwn to be being successfully 
marketed. The remaining 15 products nominated by the Blue subpanel received 
favorable votes, and the full Panel prepared dissemination-oriented 
recommendations on each one. 

In the Green subpanel 's product group, eight products were nominated 
as candidates for full-panel action, and during the plenary session three 
additional products were "championed" by individual Panel members for whom 
they held some special appeal. Thus, 11 products in this group were 
considered in a dissemination-oriented framework. Two were unaminimously 
voted "C" (hold for later review) by the full Panel. The one case that 
engendered extended discussion was seen as completely lacking in comparative 
evaluation despite the existence of a variety of comparable products. Be- 
fore bringing the product back in, the Panel pleaded, a comparative 
evaluation, independent of the developer, should be completed to show 
comparisons between this and several similar products in terms of gain 
scores. A third product was voted by seven panelists as an "F" (not 
recommended) and by the remaining three as a "D," with a number of rectify- 
ing steps to be required prior to dissemination. Although this product was 
rejected, a report on the lengthy Panel discussion is here made part of the 
record for its general value for educational research workers. 



Even though there was little if 
the achievement of stated objectives 
in the area of side effects that the 



any evidence of cognitive gains in 
for the product in question, it was 
Panel's most serious reservations 
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occurred. Cultural bias permeated the" subject content, and a number of 
examples of subtle racism were identified in the materials. Illustrations 
and art work were poorly done and in some instances markedly inappropriate. 
In retropect, tlie investment that produced this product was thought to have 
been a bad one. The minority vote, on the other hand, gave recognition to 
the fact that the product falls in a curriculum area where few other 
materials are available, and the interest among those panelists was in 
salvaging rather than rejecting the product. In this view, weaknesses 
could be removed, revisions made, and promising materials not available 
from other sources brought into being. 

The remaining eight products nominated by the Green subpanel received 
favorable votes, and ):he full Panel, as in the case of the 15 other 
"winners," prepared dissemination-oriented recommendations on each. 

Thus there were 23 produc**s in all on which dissemination recommen- 
dations were made by the full Panel. An additional grouping of 25 prod- 
ucts' was identified to be held for later review. This group was made up 
of eight judged as "holds'* by the Blue subpanel, 14 given that designation 
by the Green subpanel, plus the three that were dropped down into that 
category during the plenary Panel session. It is of particular interest 
to note that an equal number of products from the two major categories 
came into the final reckoning from each subpanel, as the following table 
shows : 





From Pool of 


From Pool of 


Panel 




Blue Subpanel 


Green Subpanel 


Decisions 










Dissemination 








Recommendation 


15 


8 


23 


Hold for Later 








Review 


9 


16. 


25 


Total 


24 


24 


48 
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Profile of the Dissemination RecoTmnendati on Group . All six NCERD specialty 
areas represented in the initial pool are represented in the recoinmended 
group. 



kj Lj \^ _L. o u- 

Area 


Number in 
Initial Pool 


Dis s emina ti on 
Recommendations 








A. Theory and Knowledge 

Building for Organizational Change 


9 


1 


B. Planning, Management, 
and Evaluation Systems 


15 


.10 


C. Instructional Personnel 
Development 


15 


1 


D. Curricular Programs 


29 


6 


E. Culturally- Targe ted 
Curricular Programs 


5 


1 


F. Home-School Intervention 
in the Early Years 


17 


4 



As to intended populations, two recommended products are targeted to 
generally disadvantaged learners and two to disadvantaged learners of 
specified kinds; six are applicable to all learners; 12 are designed to 
serve adults, including teachers in training; and one is for multiple 
populations. There are in the group three which have a manual/guide for- 
mat, one which consists of programmed material, one which is in the form 
of a game or simulation, and five which take the form primarily of tests 
or other similar measuring devices. The other 13 are in multiple formats. 
Three focus on the learning environment; nine focus on learning strategies; 
five focus on teaching methodologies; the focus of one is organizational 
efficiency; four focus on administrative techniques; and one is considered 
to have a multiple focus. 
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As to academic focus, two recommended products are in the language 
arts, one is in the performing arts, two are in mathematics, three are in 
science, one has a career-education orientation, three are multidis ciplinary 
and the remaining 11 cannot properly be described as hc/in^, an academic 
focus. Educational levels are widely dispersed, with two products for pre- 
school-kindergarten use, five for the primarN^ grades (1 through 3), two for 
senior high school (grades 10 through 12), and seven for the pos tbaccalau- 
reate level (mostly for the professional development of teachers), with the 
remaining seven having a multilevel orientation. 



Nine of the recommended products had been developed at the regional 
laboratories for educational research and development, eight had come from 
university-based research and development centers, four came from other 
groups affliated with univers ities , and two, designed for handicapped groups 
came from medical research centers. 



Appendix B shows the nine-digit alphanumeric product classification 
categories used in PROP 1972-1973 and the subcategories within each of the 
information classes. As part of this listing, the number of products in 
the initial pool of 90, together with the number in the final recommended 
pool of 23, is shown for each categor>^ of each class. 



Capsule Dissemination Recommendat -ons . In addition to this report on PROP 
19 72-19 73 , there is one which giv is the Panel 's full dissemination 
recommendation on each of the 23 exemplary products selected, together with 
extended nontechnical descriptions of those productr. Published under the 
title Dissemination Recommen d a^.ions on and Descriptions of Exemplary 
Products , this document, as in the case of the present one, is expected to 
be entered into the ERIC system and, in due course, should be available 
under an ED number. 



On the following pages are shown capsule reports in graphic form of 
the Panel's dissemination recommendations on the 23 exemplary products 
selected, together v;ith definitions of the symbols used in the graphic 
displays. For additional details on the products included in the selected 
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group, or on ihe dissemination recommendations the Panel made concerning 
them, the reader is referred to the more complete record in the published 
volume cited. Tlie form which the dissemination recommendations take in 
that volume is shown on the following page. 



ERLC 



1 



PROP 1973 U^-bl- 



PANEL REVIEW OF PRODUCTS 
for the 

National Institute of Education 



DISSEMINATION RECOMMENDATION 



Book No. PROP- AC Code No. 



Definitely 
Recommended 
(OolUr Support)"^ 



PANEL DISCUSSION 



-I UJ — I — L. 1 i i 

Conditionally 
Recommended 
(Dollar Support)"* 



Promising But 
Not Yet Ready 
(No Dollar Support)"* 



I I i I I I__L_I L 




Action Prior 
to Dissemination 
liar Support)"* 



-J I I 1 i I 



Definitely Not 
Recommended 
(No Dollar Support)** 



PANELIST COMMENTS 



NOTATION 



1 - Condition to be satisfied is indicated in Panel Discussion. 

2 - Resubmiision in 1974 is suggested. 

3 - E * Independent Evaluation or Independent Review of Available Data, T ~ Field Trials, M = More Development, O - Other Action as 

Specified. 

4 - ''DoDar Support** refers to Dissemination Dollars. 
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K D U C Al I C) N A L T E ST I N G S K R V I C K 



PRINCETON. N.J. 08540 



Area Code 609 
921-9000 
CABLE^EDLCrESrsvc 



( • 



PROP 1972-1973 
PANEL REVIEW OF PRODUCTS 
for the 

National Institute of Education 
DISSEMINATION RECOMMENDATIONS 



The 23 dissemination recommendations graphically displayed on the following 
pages are in essence the outcome of the 1972-1973 Panel Review of Products. The 
votes cf panelists are shown in the boxes, the ten divisions of each box represent- 
ing one vote each by each of the ten Panel members. Where fewer than ten of the 
Panel members voted, either because of absence or abstention, not all divisions are 
filled. Symbols used have been given the following meanings: 





Traditional Dissemination Action 




Generally Favorable Vote 


iisiiiniiiiiiiiiiiuiiiiir! 


Favorable with Reservation 
Modified Dissemination Action 




Independent Evaluation or Independent 
Review of Available Data 




Field Trials 




More Development 


im Ti 1 1 


Other Action as Specified 



Where more than one of the symbols above was applied to a given product, the 
vote has been divided appropriately under the relevant headings. Notations on the 
graphic displays may be decoded by reference to the list shown below. 

1 - Condition to be satisfied is indicated in Panel Discussion. 

2 - Resubmission in 1974 is suggested. 

3 - E = Independent Evaluation or Inf^apendent Review of Available 

Data is recommendec 
T = Field Trials are rec )minended. 
M = More Development is recommended. 
0 = Othei: Action as Specified is recommended. 

4 - "Dollar Support" refers to dissemination dollars. ^ 
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PANEL REVIEW OF PRODUCTS 
for the 

National Institute of Education 
DISSEMINATION RECOMMENDATION 



PROP- AC Code No. 

44 F-292-12-K02 



The Oral Language Program 

James L. Olivero, Robert T. Reeback, and Helgi Osterreich 

Principal Investigators 
Southwestern Cooperative Educational Laboratory 



Book No. 
36 



i I I 



Definitely 
Recommended 
(DolUr Support)"^ 



HHBHIIIIBI 

Conditionally 
Recommended 
(Dollar Support)^ 



-J L_J_J 1—1 — I — I— L. 



Promising But 
Not Yet Ready 
(No DoJIar Support)'* 



Action Prior 
to Dissemination 
(Dollar Support)^ 



_l_ 



Definitely Not 
Recommended 
(No Dollar Support)^ 



DISSEMINATION RECOMMENDATION 



Book No. ^ PROP'-AC Code No. 

62 39 D-292-62-X02 

Social Education, Grades 1-3 

Robert Randall, Principal Investigator 

Southwest Educational Development Laboratory 



t i i i ^ 



Definitely 
Recommended 
(Dollar Support)'^ 



Conditionally 
Recommended 
(Dollar Support)^ 



Promising But 
Not Yet Ready 
(No Dollar Support)^ 



Action Prior 
10 Dissemination 
(Dollar Support)^ 



I I I * i I 



Definitely Not 
Recommended 
(No Dollar Support)^ 



erJc 
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PROP 1973 



PANEL REVIEW OF PRODUCTS 
for the 

National Institute of Education 
DISSEMINATION RECOMMENDATION 



Book No, PROP- A C Code No. 

89 62 E-692-39-Z 01 

c 

Development of rfaterials for a One Year Course in African Music 

for the General Undergraduate Student 
Vada E. Butcher, Principal Investigator 
Howard University 



1 I I 



Definitely 
Recommended 
(Dollar Support 



I 



B' 



C2 



Conditionally 
Recommended 
(Dollar Support)*' 



Promising but 
Not Yet Ready 
(No Dollar Support)^ 



d3 



En 



i t I I 



-J L. 



Action Prior 
to Dissemination 
(Dollar Support)^ 



Definitely Not 
Recommended 
(No Dollar Supporl)^ 



DSSSEMINATSON RECOMMENDATION 

Book No. PROP AC (hch' No. 

51 68 . A-886-08-B01 



Instruments and Procedures for Describing Effective Teacher Behavior 
Robert C. Wilson, Principal Investigator 
University of California (Berkeley) 



■J i I 



Definitely 
Recommended 
(Dollar Support)"* 




IB 



B' 



I 1:1 L I I 



Conditionally 
Recommended 
(Dollar Support)^ 

*If D-M is done 



Promising But 
Not Yet Ready 
(No Dollar Support)"* 



M 



Action Hrior 
to Dissemination 
(Dollar Support)^ 



J L- 



Definitely Not 
Recommended 
(No Dollar SupptKt)^ 
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PANEL REVIEW OF PRODUCTS 
for the 

National Institute of Education 
DISSEMINATION RECOMMENDATION 



Book No. 
63 



PROP- AC 
72 



Cade No, 

D-692--75-Z^01 
a 



The Cluster Concept Program 

Donald Maley, Principal Investigator 

University of Maryland 



-L. 



i I 1 



Oefiniiely 
Recommended 
(Dollar Support)'* 



ill! 



I. I I 



Conditionally 
Recommended 
(Dollar Support)'* 



Promising Bui 
Not Yet Ready 
(No Dollar Support)^ 



Action Prior 
to Dissemination 
(Dollar Support)^ 



Definitely Not 
Recommended 
(No Dollar Support 



DISSEMINATION RECOMMENDATION 



Book No. PROP-- AC Code No. 

67 80 D-632-49-P01 

Individually Prescribed Instruction-Mathematics 
Robert Glaser, Principal Investigator 
Learning Research and Development Center (LRDC) 
University of Pittsburgh 



A 


B' 


C2 


1 1 1 1 1 1 1 1 L 


■ M II 1 J M !■ 

mniuiuiUiLim 


1 1 1 i 1 1 1 111 



Definitely 
Recommended 
fr^Ur Support)^ 

ERIC 



Conditionally 
Recommended 
(Dollar Support)^ 



Promising But 
Not Yet Ready 
(No Dollar Support)'* 



Action Prior 
to Dissemination 
(Dollar Support)^ 



-1 L. 



I I I t i 



Definitely Not 
Recommended 
(No Dollar Support)*^ 
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PANEL REVIEW OF PRODUCTS 
for the 

National Institute of Education 
DISSEMINATION RFCOMMFNDATION 



Book No. PR OP AC C udc No. 

41 83 F-992-12-Z^01 



The Sullivan Reading Program 

M. W. Sullivan, Principal Investigator 

Sullivan Associates 



_i ' t 1 I t l—l L_ 

Definitely 
Recommended 
(Dollar Support)'^ 



I I I I. I I I I 

IjjiLum 



Conditionally 
Recommended 
(Dollar Support)"* 



D3 



rmr 



Promising But 
Not Yet Ready 
(No Dollar Support)"* 



Action Prior 
to Dissemination 
(Dollar Support)"* 



Definitely Not 
Recommended 
(No Dollar Supp<»rt)"* 



DISSEMINATION RECOMMENDATION 



Book No. PROP -AC Code No. 

43 92 F-691-01-A01 



Home-Oriented Childhood Education Program for Rural America 
Roy W. Alford, Principal Investigator 
Appalachia Educational Laboratory 



-L-i » i » 



Definitely 
Recommended 
moiiqr Support)^ 



FRir 



Black and White Materials 
and Home Visitors 



I - J J) 11 1 I 



Conditionally 
Recommended 
(Dollar Support)"* 



Promising But 
Not Yet Ready 
(No Dollar Support)"* 



Development of Color 
Videotapes 



-I i 1 ■ I 



Action Prior 
to Disseminiition 
(Dollar Sui^f/ort)"* 



t i 



Definitely Not 
Recommended 
(No Dollar Support)"* 
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PANEL REVIEW OF PRODUCTS 
for the 

National Institute of Education 
DISSEMINATION RECOMMENDATION 



Book No. PR OP- AC Code No. 

01 96 B-881-08-U01 



Elementary School Evaluation Kit: Needs Assessment 
Ralph Hoepfner, Principal Investigator 
Center for the Study of Evaluation 



B' 



■ 1. I 



Definitely 
Recommended 
(Dollar Support)'* 



Conditionally 
Recommended 
(Dollar Support)^ 



Promising But 
Not Yet Ready 
(No Dollar Support)^ 



Action Prior 
to Dissemination 
(Dollar Support)'* 



I i I 



Definitely Not 
Recommended 
(No Dollar Support)^ 



DISSEMINATION RECOMMENDATION 



Book No. 
02 



PROP- AC 
110 



Code No. 
B-883-01-U04 



CSE-ECRC Preschool-Kindergarten Test Evaluations 
Ralph Hoepfner, Principal Investigator 
Center for the Study of Evaluation 



I I I I i I I I I 



Definitely 
Recommended 
(Dollar Support)^ 



m 


m 


..jILLL 





Gondii ionally 
Recommended 
(Dollar Support)** 



J U-l—l 1— I — I L_t- 



Promising But 
Not Yet Ready 
(No Dollar Support)'^ 



D^ 

IIINM 

Action Prior 
to Dissemination 
(Dollar Support)'^ 



' « ' ' I t t i 



Definitely Not 
Recommended 
(No Dollar Support)'* 
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PROP 1.973 



PANEL REVIEW OF PRODUCTS 
for the 

National institute of Education 
DISSEMINATION RECOMMENDATION 



Book No. PROP -AC Code No. 

03 111 B-883-92-U05 

CSE Elementary School Teat Evaluations 
Guy Strickland, Principal Investigator 
Center for the Study of the Evaluation 



1 1 rrm 



J I t I L_l I— I l_ 



A 



I I t i I 1 — i i I 

Definitely 
Recommended 
(Dollar Support)"* 



__.„.. 

iiiiiiiii 

Conditionally 
Recommended 
{Dollar Support 



_1 ' « ' I 1 

Promising But 
Not Yet Ready 
(No Doliar Support)^ 



Action Prior 
to Dissemination 
(Dollar Support)"* 



Oefinitely Not 
Recommended 
(No Dollar Support)'' 



DISSEMINATION RECOMMENDATION 



Book No. PROP 'AC Code No. 

72 130 D-699-X9-P02 

Perceptual Skills Curriculum 

Robert Glaser and Jerome Rosner, Principal Investigators 
Learning Research and Development Center (LRDC) 
University of Pittsburgh 



t ■ J I ■ 



I I 



. I 1. 



Definitely 
Recommended 
(Dollar Support)"* 



ERLC 



Conditionally 
Recommended 
(Dollar Support)^ 



Promising But 
Not Yet Ready 
(No Dollar Support)"* 



Action Prior 
lo Dissemination 
(Dollar Support)"* 



-J U 



t i i t 



Definitely Not 
Recommended 
(No Dollar Support )** 
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PANEL REVIEW OF PRODUCTS 
for the 

National Institute of Education 
DISSEMINATION RECOMMENDATION 



Book No, PROP' AC (udc No. 

04 136 B-826-08-F03 

A Sourcebook of Elementary Curricula, Programs, and Products 
Samuel N. Henrie, Principal Investigator 

Far West Laboratory for Educational Research and Development 



Definitely 
Recommended 
(Dollar SupporJ}"* 



Conditionally 
Recommended 
(DoHar Support)^ 



t i 1 i_ 



Promising But 
Not Yet Ready 
(No DoUar Support )4 



Action Prior 
to Dissemination 
(Dollar Support)^ 



1111 



i . 1 .A. 



Otffinitcly Not 
Kecommended 
(No Dollar Support 



DISSEMINATION RECOMMENDATION 



Book No. 
05 



PROP-AC 
142 



Code No. 
B-895-08-F09 



Educational Information Consultant 
Wayne Rosenoff, Principal Investigator 

Far West Laboratory for Educational Research and Development 



J '11*' 



Definitely 
Recommended 
(DolUr Support)"* 



ERIC 



Entire Product 
B' 



HmSHIE 



Conditionally 
Recommended 
(Dollar Support)"* 



L > i L t ■ i 



Audiovisual 
Component 



Promising But 
Not Yet Ready 
(No Dollar Support)"* 



Action Prior 
to Dissemination 
(Dollar Support)'* 



_l i 1 1 i \—l l_ 



Definitely Not 
Recommended 
(No Dollar Support)"* 
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PROP 197.^ 



PANEL REVIEW OF PRODUCTS 
for the 

National Institute of Education 



DISSEMINATION RECOMMENDATION 



Book No. PROP- AC Code No. 

28 144 C-896-98-F11 



Determining Instructional Pifrposes 

Joyce P. Gall and Charles Lynn Jenics, Principal Investigators 
Far West Laboratory for Educational Resfiarch and Development 



C2 



I i I I \ I 1 t 



Definitely 
Recommended 
(Dollar Support)^ 



Conditionally 
Recommended 
(Dollar Support)^ 



Promising But 
Not Yet Ready 
(No Dollar Supporc)^ 



m3m 

Action Prior 
to Dissemination 
(Dollar Support )^ 



■ 1. ± 1 1 1 i I l- ._L 
Definitely Not 

Recommended 
(No Dollar Support 



DISSEMINATION RECOMMENDATION 

No. PROP AC Code No. 

06 145 B-891-09-F12 



Early Childhood Information Unit 
Stanley Chow, Principal Investigator 

Far West Laboratory for Educational Research and Development 



■ A I i I UJ — 1 I I, 

Definitely 
Recommended 
(Dollar Support)** 



B' 



Conditionally 
Recommended 
(Dollar Support)** 



-t 1 I 



Promising But 
Not Yet Ready 
(No Dollar Support )4 



_1- 



i I i 



^Optional $ for distribution where needed 



Action Prior 
to Dissemination 
(Hollar Support)^ 



Definitely Not 
Recommended 
(No Dollar Support)** 
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PANEL REVIEW OF PRODUCTS 
for the 

National institute of Education 
DISSEMINATION RECOMMENDATION 



Book No. PROP -AC Cudv No. 

08 147 B-823-65-F14 

American Government Information Unit 
C. L. Hutchins, Principal Investigator 

Far West Laboratory for Educational Research and Development 



mmm 



Definitely 
Recommended 
(Dollar Support)"^ 



J — U-J_ 



Conditjonaliy 
Recommended 
(Dollar Support)"* 



-J — i_J I I 1 



Promising But 
Not Yet Ready 
(Nn Dollar Support)^ 



1 1 



1 I 



Action Prior 
to Dissemination 
(Dollar Support)"* 



1 I I 1 t 



Definitely No( 
Recommended 
(No Dollar Support)'* 



DISSEMINATION, REC OMMENDATION 



Book No. 
78 



PROP AC 
163 



Code No. 

D-323-99-Z 01 
P 



An Intensive Training Curriculum for Young Educable Mentally Retarded Children 
Sheila Ross> Principal Investigator 
Palo Alto Medical Research Foundation 



Definitely 
Recommended 
(Dollnr Support)"* 



Conditionally 
Recommended 
(Dollar Support)^ 



Promising But 
Not Yet Ready 
(No Dollar Support)^ 



Action Prior 
to Dissemination 
(Dollar Support)^ 



Definitely Nc 
Recommended 
(No Dollar iiupporf)"* 



-74- PROP 1973 

PANEL REVIEW OF PRODUCTS 
for the 

National Institute of Education 
DISSEMINATION RECOMMENDATION 

Book No. PROP-^AC Code No. 

12 170 B-896-08-R06 

Project Management II: Basic Principles and Techniques of Project Management 
C. Peter Cummings, Principal Investigator 
Research for Better Schools, Inc. 



D^ 



_l I V I t 



I t i 



Definitely 
Recommended 
(Dollar Support)^ 



Conditionally 
Reconiniended 
(Dollar Support)^ 



1 1 I 1 t 



Promising But 
Not Yet Ready 
(No Dollar Support)^ 



^m2m 

ActiDn Pnor 
to Dissemination 
(Dollar Support)^ 



JL_J_ 



t 1 1 t. I J 



Definitely Not 
Kecommended 
(No Dollar Support)*^ 



I 

DISSEMINATION RECOMMENDATION 

Book No, ^ PROP- AC 

13 172 

CS£/iiLP;Higher-order Cognitive, Affective, and Interpercional Skills 

Test Evaluations 
Ralph Hoepfner, Principal Investigator 
Center for the Study of Evaluation 



Code No. 
B-883-X9-U06 



i I » I II 1 I i i 

Definitely 
Recommended 
(Dollar Support)^ 



! 



mm 



Conditionally 
Recommended 
(Dollar Support)^ 



Promising Bui 
Not Yet Ready 
(No Dollar Support)^ 



[in 



TT 

J__L 



I i I 



Action Prior 
to Dissemination 
(Dollar Support)* 



Dc^nitely Not 
woinmendcd 
(Nc Dollar Support)* 
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PANEL REVIEW OF PRODUCTS 
for the 

National Institute of Education 
DISSEMINATION RECOMMENDATJON 



Book No. PROP- AC Code No. 

14 173 B-892-X8-U07 



Evaluation Workshop I: An Orientation 
Stephen P. Klein, Principal Investigator 
Center for the Study of the Evaluation 



-1—1 l_J 1_J L_ 



Definitely 
Recommended 
(Dollar Support)^ 



Cundttionally 
Recommended 
(Doller Support)^ 



Promising Uut 
Not Ye) Ready 
(iJo DolUr Supporr)"^ 



Action Prior 
Cu Disseminatinn 
(Dollar Support)"* 



Definitely Not 
Recommended 
(No D<^nar .Supp<irO'* 



DISSEMINATION RECOMMENDATION 



PROP- AC Code No. 

182 F-372-A2-Z 01 

t 

Pacemaker Games Program 

Dorothea M. Ross, Principal Investigator 
San Francisco Medical Certer 



Book No. 
48 



Definitely 
Recommended 
(Dollar Support)'* 

ERIC 



B' 



1 1^ L t 



Conditionally 
Recommended 
(Dollar Support)'^ 



tVomising But 
Nn< Yet Ready 
(No Dollar Support)"* 



D3 



-L.J 1 I I i i I L 



A.cttrtn Prior 
to Dissemination 
(Dollar Support)'* 



I » i_i till 

Definitely Not 
Recom.nenrJ* \ 
(No Dollar Support)^ 



-76- 



PROP 



PANEL REVIEW OF PRODUCTS 
for the 

National institute of Education 
DISSEMINATION RECOMMENDATION 



Book No. 
86 



PROP-AC 
187 



Cade \<>. 
D^692-69-Zj02 



Geography Curriculum Project 

Marion J. Rice, Principal Investigator 

University of Georgia 



I 1 



Definitely 
Recommended 
(Dollfftr Support)^ 



Earth: Man's Home 



i 



nil 



Conditior ^lly 
Recommf . ded 
(Dollar S' pportl** 



r 



ProTTiisinR But 
Not Yet Ready 
(No Dollar Support)** 



Action Prior 
to Dissemination 



(Dollar Support)^ 

*7,a r th : Man ' s Home 
**Other components 



Definitely Not 
Recommended 
(No Dollar Support)'* 
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Recommendations for Future 
Diss emination- Focus ed Evaluation 

Introduction , In some of the earlier parts of this report, modest efforts 
are made to apply evaluation measures to PROP — to ascertain how well it 
worked and what might be done to improve similar projects in the future. 
Here these and other indicators of success or failure and the judgmencs 
they have elicited in the direction ^f seeking improvements in educational 
product evaluation processes in th'.^ future are enumerated. 

This section is directed to NI^ stafi.^ members and others who find 
themselves with the task of planning for or devising the details of a 
project designed to compare and judge educational materials through 
application of evaluative techniques. As an integrated statement of what 
might be done to refine panel reviews of educational products in subsequent 
years, the discussion may serve just as effectively as an indicator of the 
"state of the art" of comparative evaluation, circa 19 73. Recommendations 
for minor modifications, subtle refinements, and more major changes reflect 
comments that have come from developers in the field as they worked with 
the ETS staff in assembling information on products, members of t-ie Panel 
during their meetings and thereafter and members of the ETS supporting 
staff throughout the duration of the project,-/ 

The fact that a number of recommendations for significant changes are 
made should not be construed as evidence of Panel or other participant 
dissatisfaction with the PROP mission or with the assistance the Panel 
received from supporting staff iT)eiri>ers. Rather, the two years of experience 
gained; the similarities and differences between PEP and PROP; the inter- 
actions among developers, staff, and Panel', and the insights gained through 
repeated exposure to the dynamics of developmental-product-oriented 
educational research have afforded an excellent opportunity for stock tak- 
ing, looking toward the ultimate strengthening of this process, which still 
can be regarded as being in its infancy. 
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"On Appointment of a Panel , The criterion for composition of future 
panels would be to match or excel the ones which functioned in 1971 and 
1973. They both were models of effectiveness, sage and sophisticated, 
yet compatible in their contentiousness and compassionate in their 
criticism. The black viewpoint was exceptionally effectively represented 
as was that of women, the establishment, the schoolman, the publisher, 
the college professor, the classroom teacher, and any number of other 
special-interest groups. In the future, membership should also encompass 
the Mexican- American and Puerto-Rican viewpoints, as well as those of 
other ethnic minorities. 

Use of the PROP 1972-19 73 model makes panel size a function of the 
size of the product pool. A five-member suhpanel seems to be the optimum 
size for a task force to complete initial product review, and two two-day 
meetings seem to be as much as a function such as this can be expected to 
encroach upon a busy calendar. A subpanel of this size can reasonably be 
expected to -complete initial screening of 35 to 40 educational products 
using PROP- like procedures. It would be efficacious to increase overall 
panel size in multiples of five as products are added to the initial pool 
in multiples of 40,. The outsi^Se^imit for a ten-member panel, then, would 
be 80 products and for a 15-member panel, 120. 

Another alternative to handling the *Vase load" (deviating somewhat 
from the 1972-1973 model) would be for panel members to complete a 
substantial amount of their individual product reviewing before the first , 
panel meeting. This plan is more fully described at the end of this 
section. In terms of implications for the panel, though, the alternative 
would call for provision for about four days of product review at home ia 
addition to the four days spent in meetings » plus whatever travel time is 
requi red. Using such an option, a ten-person panel should be able to 
handle a load of 100-120 products reasonably well. 

An additional alternative would be to move to a panel of 12 and sub- 
panels of four members each, with other features of the 1972-1973 PROP 
model unchani^ed. Although less o than optimum, such a plan would have put 

ERIC 



-79- 



the 19 72-1973 task of reviewing 90 products in four meeting days within 
the bounds of reason and comfort — and probably would have completely 
dissolved the pressures felt by some of the PROP panelists . 

On Composition of the Pool . The goal of future corrparative evaluations of 
products may have to be modified, since it is highly unlikely that at any 
given point in time there will be a substantial number of promising 
educational products just at the stage where they have been thoroughly 
developed and validated and still are not yet being widely disseminated. 
A future panel might well have the mission of recommending a group of 
finished products for dissemination funding and also of identifying prod- 
ucts at various late stages of development which appear to be promising 
and suggesting a variety of courses of action for them, including expend- 
ing funds for further field trials or validation; for independent 
evaluations of the products themselves or of available data on them; or 
even for further development. For products which are already at an 
advanced' and marketable stage, a panel might elect to recommend 
dissemination funding where successful distribution seems unlikely with- 
out such help. Even for successful products already being widely 
disseminated, a panel might see opportunities where additional funds might 
make it possible for distribution to be broadened into one or more unreached 
niches, such, as geographically remote parts of the country or special 
target populations • ^ 

Another variable in composing a pool is how far afielH from centers 
of e^ducational research and tievelopment one should go in search of products 
to include. A case can certainly be made for drawing upon products from 
any source whatever, as long as an irrefutable case can be made that the 
products serve to enhance and improve education in the nation's schools. 
In this case, a wider net should be cast to ensure that worthy products 
available from a variety of sources are identified and sub^jected to 
comparative evaluation. 

The suggested variations in objectives for panel reviews of products 
would require compatible variations in the way the product pools are put 
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together. Product developers should be advised of the specifics of the 
mission at the time of product submission so that they would be enabled 
to nominate products most likely to be found to be in alignment with the 
direction a particular panel review was designed to take. * 

Since one of the most significant factors contributing to the time 
pressures on 1972-1973 panelists and staff can be traced to delav in 
establishing the criteria by which products initially were to be included 
or excluded, there is a clear need for the NIE^s setting well in advance 
the ground rules to be applied. Ideally, composition of a product pool 
would be an ongoing NIE activity, and the pool for a given comparative 
evaluation would already be in existence at its outset. 

/ 

On Organization of Information . It is not sufficient to rely on information 
provided primarily by product developers in attempting to evaluate their 
products. Although developers may be able to supply data or cite studies 
which appear to provide strong evidence as to the effectiveness of their 
products, there is no guarantee that these same developers, anxious as they 
are to promote their own products, will also supply any available information 
which offers evidence which might be interpreted as negative. Partial 
rectification of this condition would result from heavier use than at 
present of the field trial as a mode for testing and producing evidence 
of success. klE should probably seek to lay greater stress on the need 
for independent evaluation and validation of educational products and for 
heavy dependence upon such verification data in dissemination- focused 
evaluation. . V . 

Whether verification is done in the dependent or independent mode, 
it is not enough to provide evidence merely as to the level of statistical 
significance of gains on test scores made by experimental, as opposed to' 
control, groups. The magnitude of the absolute gains made must be cited 
if the educational significance of the gains is to be evaluated 
intelligently. Substantial absolute gains are solid evidence of educational 
significance. Statistical significance alone may communicate very little 
indeed about educational effectiveness. 
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Products were organized into two gr'^upings for subpanel review during 
PROP 19 72-19 73 by first clustering products according to NCERD specialty 
areas and then grouping clusters into two approximately equal halves. One 
consequence of this method of organizing the Panel *s work was that all 
curriculum developments, most with expansive documentation and extensive 
materials, were assigned to the si:ne subpanel, Th. work load imposed by 
that arrangement stimulated suggested alternative thar. would call for 
putting products to be reviewed in clusters of varying nize according to 
relative complexity, with the most complex products making up a small 
cluster, relatively simple developments clustered into a lar^^eiL group, 
and products falling somewhere between assigned to a medium-sizt cluscer. 
This pattern should assure a more equal distribution of work load. 

On Orientation of the Panel . In fuuure years, it is imperative th.at the 
panel be kept fully Informed as to the procedures being followed during a 
formal review of educational products. Although the PROP panelists 
received copies of the PEP reports, they were not fully cognizant of all 
the staff support functions being carried out prior to their first meeting. 
First-time PROP panelists felt that their job would have been made easier 
had they had fuller knowledge of the procedures being followed, as, for - 
example, thv-^ submission of product precis to product developers for 
approval prior to the first I'anel meeting. 

On Establishing Criteria . PROP 19 72-19 73 led to the Panel's development 
of a tenth revision of the PEP Evaluation Worksheet and to the evolution 
of a fourth statement on selection criteria. Needless to say, each 
successive draft was felt by the Panel to be a significant improvement, 
functionally at least, over the previous one. It may be that the formu- 
lation of criteria and the capsule criterion statement on the worksheet 
form will be found to be sufficiently well honed at this noint to serve 
similar purposes in the near future without further significant modifica- 
tion. Those that have been involved in the use of these instruments 
have a reasonable degree of confidence that such will be found to b^ 
the case, at least until such time as, for example, a goal-free evaluation 
model might be tried in dissemination-focused product evaluation. 
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The 19 72-1973 experience confirmed the finding of 1971 that it is 
highly unlikely that curriculiim materials per se v/ill emerge as exemp.'.ary 
when evaluated in a broad context which includes a wide variety of ether, 
frequently highly innovative, educational materials. This limitation in 
a TROP-type activity might be rectified by the development of a special 
criterion base tailored especially to curriculum materials and by the 
application of the special criteria in a separate review process. A 
special set of selection rules should be appliei and earmarked dissemination 
funds shoult be set aside for curriculum products. Othen^ise it is 
improbable that sorely needed curricular innovations which continue to 
require en'^ouragement will emerge as recommendations from a panel review 
process . 

On PafiCl Review and Selection Procedures . Ivfhatever procedures are 
used for initial screening and review of educational products in a PRO?- 
like process, plenary panel action for final -decisions is essential. 
Both the 19 71 and 1973 panels leaned heavily on the "coinmittefi of the 
whole" as the optimum body co pool judg^ients and convert collected wisdom 
into selection-oriented decisions. Perhaps the strongest recommendation 
to be made here, then, is to continue tue emphasis on plenary panel action 
as the irreducible minimum for the final stages of product selection. 
With this constraint, another essential component is complete freedom of 
the full panel to settle upon its own decision rules and upon their applic- 
ation. Advance staff work which involves assumptions as to the panel's 
probable approach, essential as it is, may prove to be counterproductive 
and may involve some waste of time. 

When the panelists meet as a committee of the whole, they take on a 
new mission — to join forces, acting more or less as a contest jury, look- 
ing at the whole spectrum of evidence, positive and negative, considering 
what the record shows and what other panelists have learned, comparing a 
given product with competitors, evolving out of this a joint judgment or 
a set of disparate judgments, and finally making the transition from 
collective evaluation to mission-focused decision. There are an , 
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infinite number of ways to achieve this and, as experience has shown, a 
panel of the caliber of the PROP group will undoubtedly reach its own 
decisions as to how it will move from the preliminary steps to the final, 
culminating, decisive action. 

While a substantial amount of advance planning and procedural structur 
ing seems ill-advised for the final actions of a full panel, the opposite 
seems to be the case Jor the initial stages of a panel's work. Both 
effectiveness and efficiency are gained through thorough planning and 
structuring of procedures for the early stages of product review. 

Two steps will make it feasible to move a future PROP- type activity 
quickly to the initial produ»^t.-&creeni ng stage: agreement on a subpanel 
mode for initial product i.*cview, on subpanel size, and on acceptance of a 
specific evaluation worksheet for use in the early stages of work. These 
agreements would clear the way for the product review process to proceed 
with dispatch, even without a meeting at the outset. The first pass 
through an assigned pool of products could be made by the panelists 
individually at their homes. Under such a plan, the following procedures 
could apply : 

1. Panelists would be sent evaluation worksheets, details on the 
meaning of the entries, and a group of v-^o^uct precis assigned to each 
panelist's subpanel. 

2. Each panelist would rate his group of products at hoiTie over a one 
month period, sending his or her evaluation worksheets, complete with over- 
all ratings, in for summarization. 

3. At the fi it meeting, each subpanel, with summaries in hand, 
would discuss its products wich care, would refer to product dossiers as 
necessary, and would reach conclusions and draw up recommendations as to 
the kinds of dissemination-oriented action that seem appropriate on a 
product-by-product basis (the subpanel would also pinpoint, in those cases 
where additional information might be needed, what new data should be 
sought). 

4. In the two-month period between meetings, subpanel members would 
review and rate the products of a counterpart subpanel and staff members 
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would do further in-depth study on products on which information had been 
requested, 

5, ' The first day of the second meeting would be devoted to subpanel 
sessions for individual review of newly assembled infor-iaticr and refresher 
reviews by the group, product-by-product, of its own earlier conclusions 
and recommendations (subpanels at this point would not discuss prodijcts 
recommended by the othr.r subpanel). 

These preliminaries would yielu a body of inf omK^tioni conclusions, and 
recommendations for the full panel to then consider and decide upon. 
With a clear stat^iaent of mission and the preliminary judgments by the 
constituent subpanels before it, the panel could in its plenary session 
readily reach consensus on steps to ^ake in moving toward final decisions 
in fulfillment of its objectives and proceed to implement its plan, 

A possible ten-month timetable for the suggested review process is 
shown belcA/, 



July 1 Ground rules for product inclusion established; contract let 
Aug, 15 Pool fully established; dossier materials requested 
Nov, 1 Precis completed and sent to subpanels 

Dfcc. 1 Subpanelis ts ' evaluation worksheets with ratings returned 
Mid-Dec, First panel meeting 
Mid-Feb • Second panel meeting 
May 1 Reporting -completed 

Although there are numeroufj alternatives for panel review and selection 
procedures, including those suggested elsewhere in this report, the fore- 
going discussion constitutes a carefully thought-out recommendatior for the 
next effort to accomplish a dissemination-focused educational product 
evaluation. 

O n Feedback to Product Developers and to the NIE , A formalized 
mechanism should be developed whereby panelist comments and Panel ratings 
can be routinely directed to those people who can most profit from chem, 
O .otably the product devi^lopers. It is essential that such feedback also 
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be coiran^jnicated routinely to all concerned branches of the federal government. 
A syste- should be devised for identifying specific pieces of information or 
queries which should become part of the feedback effort. 

Implementation of the recommendations made here should lead to an 
evaluation process that Is likely to be of naximum use to the NIE, to the 
schools, and t:o the p.iblic both the NIE and the schools are designed to 
serve. Howevor, any future panel would have to work very hard indeed to 
equa.^ the 1972 I'"- 73 PROP Panel in conscientiousness and in effective and 
efficient accomplishments . 
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APPENDIX B 



Panel Review of Products 
PROP 1972-1973 
Product Classification Categories 



NCEKD Specialty Areas 


L- 


-000- 


-00- 


-000 


Intended Population' 


0- 


-NOO- 


-00- 


-000 


Format 


0- 


-ONO- 


-00- 


-000 


Facilitating Focus 


0- 


-OON- 


-00- 


-000 


Academic Focus 


0- 


-000- 


-NO- 


-000 


Educational Level 


0- 


-OOO- 


-ON- 


-000 


Source 


0- 


-000- 


-00- 


-LOO 


Number by Source 


0- 


-000- 


-00- 


-ONN 



PROP 1972-1973 



Initial 

NCERD Specialty Areas Pool Selections 

A, Theory and Knowledge Building 9 1 
for Organizational Change 

B, Planning, Management, and Eval- 15 10 
uation Systems 

C, Instructional Personnel Development 15 1 

D, Curricular Programs 29 ^ 

E, Culturally-Targeted Curricular 5 1 
Programs 

F, Home-School Intervention in the 17 4 
Early Years 

G, Career Education 0 0 
X. Unspecified or Undetermined 0 0 



Intended Populatio n 



0 


Not Applicable 


0 


0 


1 


Disadvantaged-Ge.ieral 


1 


2 


2 


Disadvantage'^ -bpecial Focus 


13 


2 


3 


Slow Lear-dfs 


6 


0 


4 


Nonreac'iirs 


0 


0 


5 


Academically Talented 


2 


0 


6 


All Learners 


27 


6 


7 


Parents 


0 


0 


8 


Adults 


38 


12 


9 


Multiple Population 


3 


1 


X 


Other and/or Not Specified 


0 


0 


»rmat 


and Special Approaches 






0 


Not Applicable 


0 


0 


1 


Textbooks and/or Workbooks 


2 


0 


2 


Manuals and/or Guides 


16 


3 


3 


Programmed Materials 


1 


1 


4 


Movies, Videot'ies, Slide 


1 


0 




Presentations, and/or Speaker- 








Phone Presentations 






5 


Three-Dimensional Models and/or 


1 


0 




other Materials 






6 


Computer Software 


0 


0 


7 


Games and/or Simulations 


2 


1 


8 


Tests and/or other Measurement 


9 


5 




Devices 






9 


Multiple Formats 


54 


13 


X 


Other 


0 


0 



B-3 



Facilitating Focus (Instrumentalities and Outcomes) 



0 Not Applicable 

1 Learning Environment 

2 Learning Strategy 

3 Teaching Methodology 

4 Classroom Management 

5 Organizational Efficiency 

6 Administrative Techniques 

7 Guidance and Counseling 

8 (Not Assigned) 

9 Multiple Focus 

X Other or Not Specified 



0 
6 
34 
24 
3 
3 
8 
4 
0 
5 
3 



0 
3 
9 
5 
0 
1 
4 
0 
0 
1 
0 



Academic Focus (Sub j ect-Matter Area) 



0 
1 

2 
3 

4 
5 
6 
7 



8 
9 
X 



Not Applicable 

Language Arts ar-J Literature 
(English) 

Foreign Languages and Literature 
Humanities, Fine, and Performing 
Arts 

Mathematics 
Science 

Behavioral and Social Studies 
Business Education, Vocational- 
Technical, and other Career 
Preparation 
(Not Assigned) 
Multi-Disciplinary 
Other or Not Specified 



18 

18 

2 
1 

6 

6 
12 

1 



■ 0 

14 
12 



8 
2 

0 

1 

2 
3 
0 
1 



0 

3 
3 



Educational Level 



0 Not Applicable 2 

1 Preschool or Nursery-Kindergarten 5 

2 Primary Grades or Grades 1-3 9 

3 Later Elementary or Grades 4-6 2 

4 Junior (Middle) H,S. or Grades 6-9 1 

5 Senior High School or Grades 10-12 3 

6 Junior College 0 

7 Senior College 5 

8 Postbaccalaureate 27 

9 Multi-Level 34 
X Other or Not Specified 2 



0 

2 
5 
0 
0 
2 
0 
0 
7 
7 
0 



B-4 



List of Sources 



A Appalachia Educational Laboratory, Inc. 
P.O. Box 1348 

Charleston, West Virginia 25325 

B Center for Research and Development in Higher Education 
University of Califoriiia, Berkeley 
2150 Shattuck Avenue 
Berkeley, California 94720 

C Central Midwestern Regional Educational Laboratory, Inc. 
10646 St. Charles Rock^ Road 
St. Ann, Missouri 63704 

F Far West Laboratory f jr Educational Research and Development 
1855 Folsom Street 
San Francisco, California 94133 

J Center for Social Organization of Schools 
The Johns Hopkins University 
3505 North Charles Street 
Baltimore, Maryland 21218 

K - Southwestern Cooperative Educational Laboratory, Inc. 
1404 San Mateo Boulevard, S.E. 
Albuquerque, New Jexico 87108 

M Mid-Continent Regioneil Educational Laboratory 
104 East Independence Avenue 
Kansas City, Missouri 64106 

P Learning Research and Development Center 
University of rii-uf burgh 
Pittsburgh, Pennsyxvania 15213 

R Research for Better Schools, Inc. 
Suite 1700, 1700 Market Street 
Philadelphia, Pennsylvania 19103 

T Research and Development Center for Teacher Education 
University of Texas at Austin 
Education Annex 3.1^1 
Austin, Texas 78712 



. — I 



ERIC 



B-5 



List of Sources (ContM.) 



U Center for the Study of Evaluation 

University of California, Los Angeles 
145 Moore Hall 

Los Angeles, California 90024 

W Wisconsin Research and Development Center for Cognitive Learning 
1025 West Johnson Street 
Madison, Wisconsin 53706 

X Southwest Educational Development Laboratory 
800 Brazos Street 
Austin, Texas 78701 

Z Educational Technology Centcir of Sterling Institute, Inc. 
^ 2600 Virginia Av-^nue, N, W. 
Washington, D. C. 20037 

\ Secondary School Mathematics Curriculum Improvement Study 

Teachers College 
Columbia University 
New York, New York 10027 



Z College of Fine Arts 
Howard University 
Washington, D. C. 20001 

Department of Industrial Education 
University of Maryland, College of Education 
College Park, Maryland 20742 

Z The Creative Learning Group 
^ 145 Portland Street 

Cambridge, Massachusetts 02139 

Z^ Joint Council on Economic Education 
1212 Avenue of the Americas 
New York, New York 10036 



Z Hawaii English Project 
^ University of Hawaii 
1750 Wist Place 
Honolulu, Hawaii 96822 



The Marianne Frostig Center of Educational Therapy 

5981 Venice Boulevard 

Los Angeles, California 90034 



B-6 



List of Sources (Cont'd.) 

Z. Sullivan Associates 
^ Menlo Park 
California 

Z, University of Georgia 
^ 107 Dudley Hall 

AtHv>ns, Georgia 30601 

Z Minnemast Program 

720 Washington Avenue, S.E. 
Minneapolis, Minnesota 55414 

Lincoln Filene Center for Citizenship and Public Affairs 

Tufts University 

Medford, Massachusetts 02155 

Z National Education Association 
1201 Sixteenth Street, N.W. 
Washington, D. C. 20036 

2 George Washington University Medical Center 
^ 725 Twenty-first Street, N.W. 
Washington, D. C. 20006 

Z Biological Sciences Curriculum Study 
University of Colorado 
P.O. Box 930 
Boulder, Colorado 80302 

Z Palo Alto Medical Research Foundation 
P 860 Bryant Street 

Palo Alto, California 94301 

Z Parsons State Hospital and Training Center 
^ Bureau of Child Research 
Parsons, Kansas 65357 

Z Electrical Engineering Department 
^ Stanford University 

Stanford, California 94305 

Z Lawrence Hall of Science 
University of California 
Berkeley, California 94720 

Z^ Department of Pediatrics 
School of Medicine 

University of California, San Francisco 
San Francisco, California 94122 
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APPENDIX D 
September 25, 1972 



Guidelines for Preparing Product Evaluation Precis 
General Note : 

The design of the four pages of the farrriti is iat ended to give you ^oom 
to write and revise. The space allowed for ths various headings is arbitrary; 
you need not fill it all. Be as concise as possible, without sacrificing 
completeness of important information. The review panel will have to deal 
witVi more than 100 precis at a tWf:-day F.eeting and must be cible tc niake 
comparisons readily . 

In preparing each precis, please write it in such a way that a summary 
description suitable for inclusion in the final project report can be 
extracted from it. The precis as a whole should be written in a way that 
will communicate effectively to the panel judges, while the summary state- 
ments should be written in language comprehensible to the lay reader. Please 
refer to the report entitled "Products Entered into the Pool for the Dis- 
semination Program of NCEC," submitted by ETS in June of 1971 for a complete 
record of product summaries. A sample precis from last year's PEP project 
and a sample suimnary description are attached for ready reference. 

Detailed instructions for filling out the precis form follow: 
Page 1 Accession Number 

Project Identification These will be filled in before 

Classification you receive the dossier 

Principal Investigator 

Brief Description. of Product 

Most of this should come from the descriptive information supplied by 
the developer, but it may need some supplementation. 
The description should include the following: 
Ivhat the product is 

What the product does - its purpose(s) and objective(s) and 
major characteristics (both developer-intended and incidental); 
the benefits claimed; and if possible, the way(s) in which the 
product differs from other products or programs already available. 

Whom the product is designed to serve - the group (s) for which it 
was developed and any limitations (e.g.^ "but not suitable for 
poor readers" or "useful only tor high-ability, college-bound 
students) . Groups with which the product may be used but for 
Q which it was not originally developed should be specified. 

ERIC 
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Page 2 Veriflcation-Evaluat ion/Validation 

This is the most important part of the precis. It requires the most 
rigorous attention and may eventually require the greatest amount of 
supplemental information. This section should be primarily descriptive, 
but do not skimp on detail about data, analysis, and interpretation. 

Promotionjfel brochures should be treated with caution. Hard data 
from research reports are required. The kinds of information needed 
include such things as: 

Descriptions of field tests and other evaluations 

The study designs used 

The tests used 

The data collected 

The analyses completed 

The interpretations made 

The f indings reported 

The significance attributed to the findings 
Indicate what reports on validation are available-either already in 
the dossier or requested from the principal investigator. Insofar as 
feasible deal with at least these points: 

1. The objectives of the validation study the criteria, 
formative or summative^ employed and the use to be made 
of the results . 

2. The study design and the results. Look for results in 
terms of the developer's criteria and objectives. Look 
also for side effects not anticipated. 

3. Adequacy. Comment on the scope and size of the sample(s) used; 
the scope of field testing, the efficacy of the findings, and 
the conclusion (s) . 

4. Types of tests and measurements used. Include test sites, 
target populations, and the relevance of objectives of the 
evaluation to the specified ob j ectives of the product , 

5. Replicability . Comment on the probability of being able to 
repeat the developer's validation with similar results. 




D-3 

Page 3 Implementation Implications 

Major Requirements for Implementation 

Fill in the specific implemantat ion requirements under appropriate 
headings - if there are none, simply write none . 

Personnel Required 

Indicate the number needed, special' training required (pre-service 
and in-service), availability of special "raining, and the cost of 
training and/or consultants. 

Material and Facilities 

Include both requirtid and optional items. Comment on the availability 
of the materials (e.g,, are materials available in unlimited quantities 
or are only sample materials available) and give the source from which 
they can be obtained. 

Itemize the costs and give totals for a first- time installation and 
for an ongoing prograin (on a per pupil basis, if possible). Costs to 
be considered include those for basic materials, enrichment materials, 
staff (professional, paraprof essiona 1 , technical, and clerical), pre- 
service and in-service training, consultants, repairs and replacements, 
space, and other special needs. Non-dollar costs, such as those pro- 
duced by system disruption, should also be considered. 
Administrative Considerations 

Describe the organizational implications of adopting the product, 
including such things as scheduling and classroom space. Specify the 
conditions required for installation, both on a try-out basis and for 
adoption. In what situations may the product be employed, e.g., with 
individuals only, with an entire class, department, school, or school 
system? 

Other Limitations or Factors 

This section might include comments on possible disruption of the 
existing system and possible opposition by students, staff, or community, 
/uiything that would or should influence a decision to adopt the product 
that is not already covered under other headings should be specified here. 
Try to make a careful distinction between objective data available and 
your own opinion. 
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Page 4 



Dissemination Efforts by the Developer or Publishe r 

Include both existing dissemination efforts and those planned 



for the future. 



A variety of input 



s will be necessary here. Make a careful 



analysis of what is stated in publications on the product, 
(promotional b::ochures, journal articles, dissertations, reports) 
and make inquiries of those involved, both the principal investigator 
and the publisher, if one exists. Makj some judgment as to the 
impact of the dissemination efforts already underway and the probable 
impact of efforts planned for the future. This is necessary in order 
to determine whether the product needs and can benefit from NIE- 
support. 

Description of Obstacles to Implementation ^and Suggested Dissem i na tion 
Strategies 

Be as complete as you can in citing possible problems, such as 
community opposition and the need for released time^ in implementation 
and recommend strategies for future dissemination. The material you 
supply will probably be revised or supplemented after discussion by 
the advisory panel. 
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Product Precis 



Date L'^t:. 



A'^cession No. 1st. Draft 

Revision I. 
Revision 2 
Revision 3 



Product Identification OE Number 
Title 

Principal Investigator; 

Address 



Product Classification Code 

Major Emphasis 

Target Population 

Age level . 



Brier Description of Produce 



ERIC 



Accession No. 



Evaluation/Validation 



Product Precis 



Accei3sion No. Product Precis 

Imp 1 omen tation Implications 

Major requireme n t for implementation 



Pars crenel requirements 



Materials and Facilities 



Other Administrative Considerations 



Other Factors or limitations 
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Product Dissemination Efforts 
Existing 



Planned 



Description of Obstacles to Implementation and Suggested Dissemination Strategies 
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APPENDIX F 



PRO? Action Form 
1972-1973 



Product 



Accession number^ C lassification nuTnber_ 

Team assigned^ St 



Initial review by ^on 



Second review by on_ 



Product retained in 1972 Pool: Yes No on 



NA 

ETS precis completed on^^^ by_ 



Nominated for extended study on by^ 

Selected by Panel for extended study: Yes No 

Extended information collected on _by 

Extended analysis completed on b y 

Extended analysis received on^^ by 

Product description completed on by 

Product description edited o n by 



Notes : 



APPENDIX G 



EDUCATIONAT^ TESTING SERVICE: 1>HINCET()N. N.J. 08540 



Area Code 609 
921 - 9000 
CA BLE^ED I CTESTS VC 



Memorandum For: DEVELOPERS OF PRODUCTS IN 
THE 1972-73 PROP POOL 



Subject: Descriptive Precis for Use in 

Panel Review of Products (PROP) 

From: Wesley W. Walton 

As you perhaps know, the key document available to the Panel when the Panel Review 
of Products occurs is a precis developed by the ETS professional staff (serving as 
product reviewers) on the basis cf their study of materials made available by 
developers and others concerning the individual products in our Pool . The precis 
of a product for which you are the principal investigator is enclosed. 

It may well be that the product reviewer has successfully captured the essence of 
the current situation regarding this product development, and your concurrence would 
be a most welcome response* We are even more interested, however, to learn of 
information the reviewer might have missed or interpretations that he or she may 
have made to which you would take exception. 

The Panel will meet for the first of two reviews on January 10-12. Will you 
please return well in advance of that period any comments you might wish to make 
concerning the precis, and we shall bring those comments to the attention of the 
Panel . 

In any case, a reply as soon as possible will be appreciated very much indeed. 
WWW/cw 
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